T HE G EOMETRY OF M EMORYLESS S TOCHASTIC P OLICY O PTIMIZATION IN I NFINITE -H ORIZON POMDP S
Anonymous authors
Paper under double-blind review
A BSTRACT
We consider the problem of ﬁnding the best memoryless stochastic policy for an inﬁnite-horizon partially observable Markov decision process (POMDP) with ﬁnite state and action spaces with respect to either the discounted or mean reward criterion.
We show that the (discounted) state-action frequencies and the expected cumulative reward are rational functions of the policy, whereby the degree is determined by the degree of partial observability.
We then describe the optimization problem as a linear optimization problem in the space of feasible state-action frequencies subject to polynomial constraints that we characterize explicitly.
This allows us to address the combinatorial and geometric complexity of the optimization problem using recent tools from polynomial optimization.
In particular, we estimate the number of critical points and use our polynomial programming description of reward maximization to solve a navigation problem in a grid world.
I NTRODUCTION
Markov decision processes (MDPs) were introduced by Bellman (1957) as a model for sequential decision making and optimal planning (see, e.g., Howard, 1960; Derman, 1970; Puterman, 2014).
Many algorithms in reinforcement learning rely on the ideas and methods developed in the context of
MDPs (see, e.g., Sutton & Barto, 2018).
Often in practice, the decisions need to be made based only on incomplete information of the state of the system.
This setting is modeled by partially observable
Markov decision processes (POMDPs) introduced by ˚Astr ¨ om (1965) (for a historical discussion see
Monahan, 1982), which have become an important model for planning under uncertainty.
In this work we pursue a geometric characterization of the policy optimization problem in POMDPs over the class of stochastic memoryless policies and its dependence on the degree of partial observability.
It is well known that acting optimally in POMDPs may require memory ( ˚Astr ¨ om, 1965).
A POMDP with unlimited memory policies can be modeled as a belief state MDP, where the states are replaced by probability distributions that serve as sufﬁcient statistics for the previous observations (Kaelbling et al., 1998; Murphy, 2000).
Finding an optimal policy in this class is PSPACE-complete for ﬁnite horizons (Papadimitriou & Tsitsiklis, 1987) and undecidable for inﬁnite horizons (Madani et al., 2003; Chatterjee et al., 2016).
Therefore, it is of interest to consider POMDPs with constrained policy classes.
A natural class to consider are memoryless policies, also known as reactive or Markov policies, which select actions based solely on the current observations.
In this case, it is useful to allow the actions to be selected stochastically, which not only allows for better solutions but also provides a continuous optimization domain (Singh et al., 1994).
Although they are more restrictive than policies with memory, memoryless policies are attractive as they are easier to optimize and are versatile enough for certain applications (Tesauro, 1995; Loch &
Singh, 1998; Williams & Singh, 1999; Kober et al., 2013).
In fact, ﬁnite-memory policies can be modeled in terms of memoryless policies by supplementing the state of the system with an external memory (Littman, 1993; Peshkin et al., 1999; Icarte et al., 2021).
Hence theoretical advances on memoryless policy optimization are also of interest to ﬁnite-memory policy optimization.
Theoretical aspects and optimization strategies over the class of memoryless policies have been studied in numerous works (see, e.g., Littman, 1994; Singh et al., 1994; Jaakkola et al., 1995; Loch & Singh, 1998; Williams & Singh, 1999; Baxter et al., 2000; Baxter & Bartlett, 2001; Li et al., 2011; Azizzadenesheli et al., 2018).
However, ﬁnding exact or approximate optimal memoryless stochastic policies for POMDPs is still considered an open problem (Azizzadenesheli et al., 2016), which is
NP-hard in general (Vlassis et al., 2012).
One reason for the difﬁculties is that, even in a tabular setting, the problem is non-convex and can exhibit suboptimal strict local optima (Bhandari & Russo, 2019).
For memoryless policies the expected cumulative reward is a linear function of the corresponding (discounted) state-action frequencies.
In the case of MDPs the feasible set of state-action frequencies is known to form a polytope, so that the optimization problem can be reduced to a linear program (Manne, 1960; De Ghellinck, 1960; d’Epenoux, 1963; Hordijk & Kallenberg, 1981).
On the other hand, to the best of our knowledge, for POMDPs the speciﬁc structure of the feasibility constraints and the optimization problem have not been studied, at least not in the same level of detail (see related works below).
Related works In MDPs, the (discounted) state-action frequencies have been shown to form a polytope resp.
a compact convex set in the ﬁnite resp.
countable state-action cases, whereby the extreme points are given by the state-action frequencies of deterministic stationary policies (Derman, 1970; Altman & Shwartz, 1991).
Further, Dadashi et al.
(2019) showed that the set of state value functions in ﬁnite state-action MDPs is a ﬁnite union of polytopes.
The set of stationary state-action distributions of POMDPs has been studied by Mont´ufar et al.
(2015) highlighting a decomposition into inﬁnitely many convex subsets whose dimensions depend on the degree of observability.
Although this decomposition can be used to localize optimal policies to some extent, a description of the pieces in combination is still missing, needed to capture the properties of the optimization problem.
We will obtain a detailed description in terms of ﬁnitely many polynomial constraints with closed form expressions and bound their degrees in terms of the observation mechanism.
This yields the formulation of reward maximization in inﬁnite-horizon POMDPs as a polynomial optimization problem, where Cohen & Parmentier (2018) obtained a description of ﬁnite-horizon POMDPs as a mixed integer linear program.
Grinberg & Precup (2013) showed that the expected mean reward is a rational function and obtained bounds on the degree of this function.
We generalize this to the setting of discounted rewards and reﬁne the result by relating the rational degree to the degree of observability.
For both MDPs and
POMDPs the expected cumulative reward is known to be a non-convex function of the policy even for tabular policy models (Bhandari & Russo, 2019).
Nonetheless, for MDPs critical points can be shown to be global maxima under mild conditions.
In contrast, for POMDPs or MDPs with linearly restricted policy models, it is known that non-global local optimizers can exist (Baxter et al., 2000;
Poupart et al., 2011; Bhandari & Russo, 2019).
However, nothing is known about the number of local optimizers.
We will present bounds on the number of critical points building on our computation of the rational degree and feasibility constraints.
The structure of the expected cumulative reward has been studied in terms of the location of the global optimizers and the existence of local optimizers.
Most notably, it is well known that in MDPs there always exist optimal policies which are memoryless and deterministic (see Puterman, 2014).
In the case of POMDPs, optimal memoryless policies may need to be stochastic (see Singh et al., 1994).
Mont ´ ufar et al.
; Mont ´ ufar & Rauh (2017); Mont ´ ufar et al.
(2019) obtained upper bounds on the number of actions that need to be randomized by these policies, which in the worst case is equal to the number of states that are compatible with the observation.
Although we do not improve these results (which are indeed tight in some cases), our description of the expected cumulative reward function leads to a simpler proof of the bounds obtained by Mont´ufar et al.
Neyman (2003) considered stochastic games as semialgebraic problems showing that the minmax and maxmin payoffs in an n -player game are semialgebraic functions of the discount factor.
Although this is not directly related to our work, we take a similar philosophy.
We pursue a semialgebraic description of the feasible set of discounted state-action frequencies in POMDPs, which is closely related to the general spirit of semialgebraic statistics (Zwiernik, 2016).
Based on this we characterize the properties of the optimization problem by its algebraic degree, a concept that has been advanced in recent works on polynomial optimization (Bajaj, 1988; Catanese et al., 2006; Nie & Ranestad, 2009; Draisma et al., 2016; ¨Ozl¨um C¸elik et al., 2021).
Contributions We obtain results for inﬁnite-horizon POMDPs with memoryless stochastic policies under the mean or discounted reward criteria which can be summarized as follows.
We show that the state-action frequencies and the expected cumulative reward can be written as fractions of determinantal polynomials in the entries of the stochastic policy matrix.
We show that the degree of these polynomials is directly related to the degree of observability (see Theorem 4).
We describe the set of feasible state-action frequencies as a basic semialgebraic set, i.e., as the solution set to a system of polynomial equations and inequalities, for which we also derive closed form expressions (see Theorem 16 and Remark 18).
We reformulate the expected cumulative reward optimization problem as the optimization of a linear function subject to polynomial constraints (see Remark 19), which we use to solve a navigation problem in a grid world (see Appendix F).
This is a POMDP generalization of the dual linear programming formulation of MDPs (Kallenberg, 1994; Puterman, 2014).
We present two methods for computing the number of critical points, which rely, respectively, on the rational degree of the expected cumulative reward function and the geometric description of the feasible set of state-action frequencies (see Theorem 20, Proposition 21 and Appendix D)
P RELIMINARIES
We use standard notation for probability distributions and Markov kernels.
We denote the simplex of probability distributions on a ﬁnite set X by ∆ X .
An element µ ∈ ∆ X is a vector in R X with non-negative entries µ x = µ ( x ) , x ∈ X adding to one.
We denote the set of Markov kernels from a ﬁnite set X to another ﬁnite set Y by ∆ XY .
An element Q ∈ ∆ XY is a |X| × |Y| row stochastic matrix with entries Q xy =
Q ( y | x ) , x ∈ X , y ∈ Y .
Given Q (1) ∈ ∆ XY and Q (2) ∈
∆ YZ we denote their composition into a kernel from X to Z by Q (2) ◦ Q (1) ∈ ∆ XZ .
Given p ∈ ∆ X and Q ∈ ∆ XY we denote their composition into a joint probability distribution by p ∗ Q ∈ ∆ X×Y , ( p ∗ Q )( x, y ) :
= p ( x ) Q ( y | x ) .
The support of a vector v ∈ R X is the set supp( v ) = { x ∈ X : v x (cid:54) = 0 } .
A partially observable Markov decision process or shortly POMDP is a tuple ( S , O , A , α, β, r ) .
We assume that S , O and A are ﬁnite sets which we call the state , the observation and the action space respectively.
We ﬁx a Markov kernel α ∈ ∆ S×AS which we call the transition mechanism and a kernel β ∈ ∆ SO which we call the observation mechanism .
Further, we consider an instantaneous reward vector r ∈ R S×A .
We call the system fully observable if β = id 1 , in which case the POMDP simpliﬁes to a Markov decision process or shortly MDP .
As policies we consider elements π ∈ ∆ OA .
Every policy π ∈ ∆ OA deﬁnes a Markov kernel τ = τ π = π ◦ β ∈ ∆ SA which we call the effective policy .
Every policy deﬁnes transition kernels P π ∈ ∆ S×AS×A and p π ∈ ∆ SS through
[Equation]
For any initial state distribution µ ∈ ∆ S , a policy π ∈ ∆ OA deﬁnes a Markov process on S × A with transition kernel P π which we denote by P π,µ .
For a discount rate γ ∈ (0 , 1) and γ = 1 we deﬁne
R µ γ ( π ) :
=
E P π,µ (cid:34) (
[Equation]
) ∞ (cid:88) t
= γ t r ( s t , a t ) (cid:35) and R µ 1 ( π ) :=
lim T
[Equation]
P
[Equation]
µ (cid:34)
T
[Equation]
(cid:88) t
= r
( s
[Equation]
a t ) (cid:35) , called the expected discounted reward and the expected mean reward , respectively.
For a policy π we deﬁne the value function V πγ ∈ R S via V πγ ( s ) :
= R δ s γ ( π ) , s ∈ S , where δ s is the Dirac distribution concentrated at s .
A short calculation (Zahavy et al., 2021) shows that
[Equation]
where
[Equation]
Here η π,µγ is an element of ∆ S×A called the expected (discounted) state-action frequency (Derman, 1970), (discounted) visitation/occupancy measure or on-policy distribution (Sutton & Barto, 2018).
Denoting the state marginal of η π,µγ by ρ π,µγ ∈ ∆ S , have η π,µγ ( s, a )
=
ρ π,µγ ( s )( π ◦ β )( a | s ) .
We recall the following well-known facts.
Proposition 1 (Existence of state-action frequencies and rewards) .
Let ( S , O , A , α, β, r ) be a
POMDP, γ ∈ (0 , 1] and µ ∈ ∆ S .
Then η π,µγ , ρ π,µγ and R µγ ( π ) exist for every π ∈ ∆ OA and µ ∈ ∆ S and are continuous in γ ∈ (0 , 1] for ﬁxed π and µ .
For γ = 1 we work under the following standard assumption in the (PO)MDP literature 2 .
Assumption 2 (Uniqueness of stationary disitributions) .
If γ = 1 , we assume that for any policy π ∈ ∆ OA there exists a unique stationary distribution η ∈ ∆ S×A of P π .
The following proposition shows in particular that for any initial distribution µ , the inﬁnite time horizon state-action frequency η π,µγ is the unique discounted stationary distribution of P π .
Proposition 3 (State-action frequencies are discounted stationary) .
Let ( S , O , A , α, β, r ) be a
POMDP, γ ∈ (0 , 1] and µ ∈ ∆ S .
Then η π,µγ is the unique element in ∆ S×A satisfying η π,µγ
[Equation]
Further, ρ π,µγ is the unique element in ∆ S satisfying ρ π,µγ
[Equation]
Since the state-action frequencies satisfy this generalized stationarity condition, we sometimes refer to them as discounted stationary distributions and denote the set of all state-action frequencies in the fully and in the partially observable case by
[Equation]
We have seen that the expected cumulative reward function R µγ : ∆ OA
→ R factorises according to
[Equation]
This is illustrated in Figure 1.
We make use of this decomposition in two different ways.
First, in Section 3 we study the algebraic properties of the parametrization Ψ µγ of the set of state-action frequencies N µ,βγ .
In Section 4 we derive a description of N µ,βγ via polynomial inequalities.
T HE P ARAMETRIZATION OF D ISCOUNTED S TATE -A CTION F REQUENCIES
In this section we show that in POMDPs the discounted state-action frequencies, the value function and the expected cumulative reward are rational functions and relate their rational degree, which can be interpreted as a measure of their complexity, to the degree of observability.
Here, we say that a function is a rational function of degree at most k if it is the fraction of two polynomials of degree at most k .
First note that by Cramer’s rule (see details in Appendix B.2), it holds that
[Equation]
where ( I − γp Tπ ) µs denotes the matrix obtained by replacing the s -row of I − γp Tπ with µ .
Since p π is linear in π and the determinant is a polynomial, this is a rational function in the entries of π .
For the degree, we show the following result in Appendix B.1.
Theorem 4 (Degree of POMDPs) .
Let ( S , O , A , α, β, r ) be a POMDP, µ ∈ ∆ S be an initial distribution and γ ∈ (0 , 1) a discount factor.
The state-action frequencies η π,µγ and ρ π,µγ , the value function V πγ and the expected cumulative reward R µγ ( π ) are rational functions with common denominator in the entries of the policy π .
Further, if they are restricted to the subset Π ⊆ ∆ OA of policies which agree with a ﬁxed policy π 0 on all states outside of
O ⊆ O , they have degree at most
[Equation]
Hence the number of states that are compatible with o determines the complexity of the discounted state-action frequencies, the value function and the reward function.
Various reﬁnements of the theorem are presented in Appendix B.1.
For the mean reward case and under an ergodicity assumption, Grinberg & Precup (2013) showed that the stationary distributions are a rational function of degree of most |S| of the policy.
From Theorem 4 we can derive multiple implications (see also
Appendix D.5.2 for implications on the optimization landscape):
[Figure]
Observation policies
State policies
[Figure]
frequencies
[Figure]
Corollary 5 (Feasible state-action frequencies and value functions form semialgebraic sets) .
Consider a POMDP ( S , O , A , α, β, r ) and let µ ∈ ∆ S be an initial distribution and γ ∈ (0 , 1) a discount factor.
The set of discounted state-action frequencies and the set of value functions are semialgebraic sets 3 .
Proof.
By Theorem 4, both sets possess a rational and thus a semialgebraic parametrization and are semialgebraic by the Tarski-Seidenberg theorem (Neyman, 2003).
We compute the deﬁning linear and polynomial (in)equalities of the set of feasible state-action frequencies in Section 4 for MDPs and POMDPs respectively, which shows in particular that also in the mean case the state-action frequencies form a semialgebraic set.
The special properties of degree-one rational functions, which we elaborate in the Appendix B.3, imply the following results.
The ﬁrst one is a reﬁnement of Dadashi et al.
(2019, Lemma 4), stating that linear interpolation between two policies that differ on a single state leads to a linear interpolation of the corresponding value functions.
We generalize this to state-action frequencies, explicitly compute the interpolation speed and describe the curves obtained by interpolation between arbitrary policies.
Further, our formulation extends to the mean reward case (see Remark 43).
Proposition 6.
Let ( S , A , α, r ) be a MDP and γ ∈ (0 , 1) .
Further, let π 0 , π 1 ∈ ∆ SA be two policies that differ on at most k states.
For any λ ∈ [0 , 1] let V λ ∈ R S and η µλ ∈ ∆ S×A denote the value function and state-action frequency belonging to the policy π
+ λ ( π 1 − π 0 ) with respect to the discount factor γ , the initial distribution µ and the instantaneous reward r .
Then the rational degrees of λ (cid:55)→ V λ and λ (cid:55)→ η λ are at most k .
If they differ on at most one state ˜ s ∈ S then
V λ
= V
+ c ( λ ) · ( V 1 − V 0 ) and η µ λ =
η µ
+ c ( λ ) · ( η µ 1 − η µ 0 ) for all λ ∈ [0 , 1] , where
[Equation]
In particular, for a blind controller with two actions the set of feasible value functions and the set of feasible state-action frequencies are pieces of curves with rational parametrization of degree at most k = |S| .
By Theorem 4, the cumulative reward of (PO)MDPs is a degree-one rational function in every row of the (effective) policy.
Since degree-one rational functions attain their maximum in a vertex (Corollary 39), we immediately obtain the existence of an optimal policy which is deterministic on every observation from which the state can be reconstructed, which has been shown using other methods by Mont´ufar et al.
Proposition 7 (Determinism of optimal policies) .
Let ( S , O , A , α, β, r ) be a POMDP, µ ∈ ∆ S be an initial distribution and γ ∈ (0 , 1) a discount factor and let π ∈ ∆ OA be an arbitrary policy and denote the set of observations o such that |{ s ∈ S | β
( o | s ) > 0 }| ≤ 1 by O .
Then there is a policy ˜ π , which is deterministic on every o ∈ O such that R µγ (˜ π ) ≥ R µγ ( π ) .
Proof.
For o ∈ O , the reward function restricted to the o -th component of the policy is a rational function of degree at most one.
By Corollary 39 (see Appendix B.3.2), there is a policy ˜ π , which is deterministic on o and satisﬁes R µγ (˜ π ) ≥ R µγ ( π ) .
Iterating over o ∈
O yields the result.
On observations which can be made from more than one state, bounds on the required stochasticity were established by Mont´ufar & Rauh (2017); Mont´ufar et al.
T HE S ET OF F EASIBLE D ISCOUNTED S TATE -A CTION F REQUENCIES
In Corollary 5, we have seen that the state-action frequencies form a semialgebraic set.
Now we aim to describe its deﬁning polynomial inequalities.
In the case of full observability, the feasible stateaction freqencies are known to form a polytope (Derman, 1970; Altman & Shwartz, 1991) which is closely linked to the dual linear programming formulation of MDPs (Hordijk & Kallenberg, 1981), see also Figure 1.
We ﬁrst describe the combinatorial properties of this polytope and then extend the result to the partially observable case, for which we obtain explicit polynomial inequalities induced by the partial observability under a mild assumption.
Most proofs are postponed to Appendix C. In
Section 5 we discuss how the degree of these deﬁning polynomials allows us to upper bound the number of critical points of the optimization problem.
We use the following explicit version of the classic result characterization of the state-action frequencies as a polytope (see Appendix C.1).
Proposition 8 (Characterization of N µγ ) .
Let ( S , A , α, r ) be a MDP, µ ∈ ∆ S be an initial distribution and γ ∈ (0 , 1] .
It holds that
[Equation]
s
[Equation]
where w sγ := δ s ⊗ 1 − γα ( s |· , · ) .
For γ ∈ (0 , 1) , ∆ S×A can be replaced by [0 , ∞ ) S×A in (3) .
Now we turn towards the partially observable case and introduce the following notation.
Deﬁnition 9 (Effective policy polytope) .
We call the set of effective policies τ
= π ◦ β ∈ ∆ SA the effective policy polytope and denote it by ∆ S ,β A .
Note that ∆ S ,β A is indeed a polytope since it is the image of the polytope ∆ OA under the linear mapping
π (cid:55)→ π ◦ β .
Hence, we can write it as an intersection ∆ S ,β A = ∆ SA ∩ U ∩ C , where
U , C ⊆ R S×A are an afﬁne subspace and a polyhedral cone and describe a ﬁnite set of linear equalities and a ﬁnite set of linear inequalities respectively.
Deﬁning linear inequalities of the effective policy polytope Obtaining inequality descriptions of the images of polytopes under linear maps is a fundamental problem that is non-trivial in general.
It can be approached algorithmically, e.g., by Fourier-Motzkin elimination, block elimination, vertex approaches, and equality set projection (Jones et al., 2004).
In the special case where the linear map is injective, one can give the deﬁning inequalities in closed form as we show in Appendix C.2.1.
Hence, for the purpose of obtaining closed-formulas for the effective policy polytope we make the following assumption.
However, our subsequent analysis in Section 4 can handle any inequalities.
Assumption 10.
The matrix β ∈ ∆ SO ⊆
R S×O has linearly independent columns.
Remark 11.
The assumption above does not imply that the system is fully observable.
Recall that if β has linearly independent columns, the Moore-Penrose takes the form β + = ( β T β ) − 1 β T .
An interesting special case is when β is deterministic but may map several states to the same observation (this is the partially observed setting considered in numerous works).
In this case, β + = diag( n − 1 1 , . .
n − 1 |O| )
β T , where n o denotes the number of states with observation o .
In this case, β + so agrees with the conditional distribution
β
( s | o ) with respect to a uniform prior over the states; however, this is not in general the case since β + can have negative entries.
Theorem 12 ( H -description of the effective policy polytope) .
Let ( S , O , A , α, β, r ) be a POMDP and let Assumption 10 hold.
Then it holds that
[Equation]
where U = { π ◦ β | π ∈ R S×O } is a subspace,
C = { τ ∈ R S×A | β + τ ≥ 0 } is a pointed polyhedral cone and D = { τ ∈ R S×A | (cid:80) a ( β + τ ) oa = 1 for all o ∈ O} an afﬁne subspace.
Further, the face lattices of ∆ OA and ∆ S ,β A are isomorphic.
Deﬁning polynomial inequalities of the feasible state-action frequencies In order to transfer inequalities in ∆ SA to inequalities in the set of state-action frequencies N µγ , we use that the inverse of π (cid:55)→ η π is given through conditioning (see Proposition 46) under the following assumption.
Assumption 13 (Positivity) .
Let ρ π,µγ > 0 hold entrywise for all policies π ∈ ∆ SA .
Note that this positivity assumption holds in particular, if either α > 0 and γ > 0 or γ < 1 and µ > 0 or entrywise (see Appendix C.1).
Using this conditioning principle, we can translate linear inequalities in ∆ SA into polynomial inequalities in N µγ .
More precisely, we have the following result.
Proposition 14 (Correspondence of inequalities) .
Let ( S , A , α, r ) be a MDP, τ ∈ ∆ SA and let η ∈ ∆ S×A denote its corresponding discounted state-action frequency for some µ ∈ ∆ S and γ ∈ (0 , 1] .
Let c ∈ R , b ∈ R S×A and set S :
= { s ∈ S | b sa (cid:54) =
for some a ∈ A} .
Then (cid:88) s,a b sa τ sa ≥ c implies (cid:88) s ∈ S (cid:88) a b sa η sa (cid:89) s (cid:48)
∈ S \{ s } (cid:88) a (cid:48)
η s (cid:48) a (cid:48) − c (cid:89) s (cid:48) ∈ S (cid:88) a (cid:48)
η s (cid:48) a (cid:48) ≥ 0 , where the right is a multi-homogeneous polynomial 4 in the blocks ( η sa ) a ∈A ∈ R A with multi-degree
S ∈ N S .
If further Assumption 13 holds, the inverse implication also holds.
The preceding proposition shows that the state-action frequencies of a linearly constrained policy model, where the constraints only address the policy in individual states form a polytope.
However, the effective policy polytope is almost never of this box type (see Remark 55).
Example 15 (Blind controller) .
For a blind controller the linear equalities deﬁning the effective policy polytope in ∆ S A are τ s 1 a − τ s 2 a
= τ ( a | s 1 ) − τ ( a | s 2 ) =
for all a ∈ A , s 1 , s 2 ∈ S .
They translate into the polynomial equalities η s 1 a ρ s 2 − η s 2 a ρ s 1 = 0 for all a ∈ A , s 1 , s 2 ∈ S .
In the case that A = { a 1 , a 2 } , we obtain
[Equation]
η s 2 a 1 + η s 2 a 1 ) − η s 2 a 1 ( η s 1 a 1 + η s 1 a 1 ) =
η s 1 a 1 η s 2 a 2 − η s 1 a 2 η s 2 afor all s 1 , s 2 ∈ S , which is precisely the condition that all 2 × 2 minors of η vanish.
Hence, in this case the set of state-action frequencies N µ,βγ is given as the intersection of N µγ of state-action frequencies of the associated MDP and the determinantal variety D 1 of rank one matrices.
Now we present a general result concerning the semialgebraic structure of the set of feasible stateaction frequencies.
Theorem 16.
Let ( S , O , A , α, β, r ) be a POMDP, µ ∈ ∆ S and γ ∈ (0 , 1] and assume that Assumption 13 holds.
Then we have N µ,βγ =
N µγ ∩ V ∩ B , where V is a variety described by multi-homogeneous polynomial equations and B is a basic semialgebraic set described by multihomogeneous polynomial inequalities.
Further, the face lattices of ∆ S ,β A and N µ,βγ are isomorphic.
Remark 17.
The variety V corresponds to the subspace U and the basic semialgebraic set B to the cone C from (4).
Further, closed form expressions for the deﬁning polynomials can be computed using Proposition 14 (see also Remark 18).
The statement about isomorphic face lattices is in the sense that ∆ S ,β A and N µ,βγ have the same number of surfaces of a given dimension with the same neighboring properties.
This can be seen in Figure 1, where the effective policy polytope and the set of state-action frequencies both have four vertices, four edges, and one two-dimensional face.
Remark 18.
By Theorem 12 and Proposition 14, the deﬁning polynomials of the basic semialgebraic set B from Theorem 16 are indexed by a ∈ A , o ∈ O and are given by
[Equation]
where S o := { s ∈ S | β + os (cid:54) = 0 } .
The polynomials depend only on β and not on γ , µ nor α , and have | S o ||A|
|S
o |− 1 monomials of degree | S
o | of the form (cid:81) s
∈ S
o η sf ( s ) for some f : S o → A .
In particular, we can read of the multi-degree of p ao with respect to the blocks ( η sa ) a ∈A which is given by 1 S o (see also Proposition 14).
A complete description of the set N µ,βγ via (in)equalities follows from the description of N µγ via linear (in)equalities given in (3).
In Section 5 we discuss how the degree of these polynomials controls the complexity of the optimization problem.
Remark 19 (Planning in POMDPs as a polynomial optimization problem) .
The semialgebraic description of the set N µ,βγ of feasible state-action distributions allows us to reformulate the reward maximization as a polynomially constrained optimization problem with linear objective (see also
Remark 58 and Algorithm 1).
This reformulation allows the use of polynomial optimization algorithms, which we demonstrate in Appendix F on the toy example of Figure 1 and a grid world.
N UMBER AND L OCATION OF C RITICAL P OINTS
Although the reward function of MDPs is non convex, it still exhibits desirable properties from a standpoint of optimization.
For example, without any assumptions, every policy can be continuously connected to an optimal policy by a path along which the reward is monotone (see Appendix D.5).
Under mild conditions, all policies which are critical points of the reward function are globally optimal (Bhandari & Russo, 2019).
In partially observable systems, the situation is fundamentally different.
In this case, suboptimal local optima of the reward function can exist as can be seen in Figure 1 (see also Poupart et al., 2011;
Bhandari & Russo, 2019).
In the following we use the geometry description of the discounted state-action frequencies to bound the number and location of critical points.
These are important properties of the optimization problem and have implications on the required stochasticity of optimal policies.
In Appendix D we discuss the mean reward case and an example.
In Appendix D.5.2 we describe the sublevelsets as semialgebraic sets.
We regard the reward as a linear function p 0 over the set of feasible state-action frequencies N µ,βγ .
Since under Assumption 13 π (cid:55)→ η π is injective and has a full-rank Jacobian everywhere (see
Appendix C.1.1), the critical points in the policy polytope ∆ OA correspond to the critical points of p 0 on N µ,βγ (see Trager et al., 2019).
In general, critical points of this linear function can occur on every face of the semialgebraic set N µ,βγ .
The optimization problem thus has a combinatorial and a geometric component, corresponding to the number of faces of each dimension and the number of critical points in the relative interior of any given face.
We have discussed the combinatorial part in
Theorem 16 and focus now on the geometric part.
Writing N µ,βγ = { η ∈ R S×A | p i ( η ) ≤ 0 ,
i ∈ I } , we are interested in the number of critical points on the interior of a face,
[Equation]
Note that a point η ∈ int( F J ) is critical, if and only if it is a critical point on the variety
[Equation]
For the sake of notation we write J = { 1 , . . .
, m } .
We can bound the number of critical points in the interior of the face by the number of critical points of the polynomial optimization problem for p 0 ( η ) subject to p 1 ( η ) = · · · = p m ( η ) = 0 .
This number is upper bounded by the algebraic degree of the problem which controls also the (algebraic) complexity of optimal policies (see Appendix D. for details).
Using Theorem 12, Proposition 14 and an upper bound on the algebraic degree of polynomial optimization by Nie & Ranestad (2009) yields the following result.
Theorem 20.
Consider a POMDP ( S , O , A , α, β, r ) , γ ∈ (0 , 1) , assume that r is generic, that β ∈ R S×O is invertible, and that Assumption 13 holds.
For any given I ⊆
A × O consider the following set of policies, which is the relative interior of a face of the policy polytope:
[Equation]
Let O :=
{ o ∈ O | ( a, o ) ∈
I for some a } and set k
o := |{ a | ( a, o ) ∈ I }| as well as d o := |{ s | β + os (cid:54) = 0 }| .
Then, the number of critical points of the reward function on int( F ) is at most
[Equation]
where l = |S| ( |A| − 1) − | I | .
If α and µ are generic, this bound can be reﬁned by computing the polar degrees of multi-homogeneous varieties (see Proposition 21 for a special case).
The same bound holds in the mean reward case γ = 1 for l given in Remark 61.
By results from Mont´ufar & Rauh (2017) a POMDP has optimal memoryless stochastic policies with | supp π ( ·| o ) | ≤ l o , where l o = | supp β
( o |· ) | ≥ 1 .
Hence, we may restrict attention to optimization over N µ,βγ with k = (cid:80) o ∈O k o active inequalities (zeros in the policy), where k o = max {|A|− l o , 0 } .
Over these faces of the feasible set, the algebraic degree of the reward maximization problem is upper bounded by (cid:81)
[Table]
( d
[Table] [Table]
due to Theorem 20.
In the special case of MDPs the bound shows that for MDPs only deterministic policies can be critical points of the reward function (see Corollary 62).
Setting I := ∅ shows that there are no critical points in the interior of the policy polytope ∆ OA .
This requires the assumption that β is invertible (see Appendix D.4).
The bound in Theorem 20 neglects the speciﬁc algebraic structure of the problem, and can be reﬁned by considering polar degrees of determinantal varieties.
This yields the following tighter upper bound for a blind controller with two actions (see Appendix D.3).
Proposition 21 (Number of critical points in a blind controller) .
Let ( S , O , A , α, β, r ) be a POMDP describing a blind controller with two actions, i.e., O = { o } and A = { a 1 , a 2 } and let r, α and µ be generic and let γ ∈ (0 , 1) .
Then the reward function R µγ has at most |S| critical points in the interior int(∆ OA ) ∼ = (0 , 1) of the policy polytope and hence at most |S| + 2 critical points.
In Appendix D.4 we provide examples of blind controllers which have several critical points in the interior (0 , 1) ∼ = int(∆ OA ) and strict maxima at the two endpoints of the interval [0 , 1] ∼ = ∆ OA respectively.
Such points are called smooth and non-smooth critical points respectively.
C ONCLUSION
We described the geometry and algebraic properties of POMDPs and related the rational degree of the discounted state-action frequencies and the expected cumulative reward function to the degree of observability.
We described the set of feasible state-action frequencies as a basic semialgebraic set, computed explicit expressions for the deﬁning polynomials.
In particular, this yields a polynomial programming formulation of POMDPs extending the linear programming formulation of MDPs.
Based on this and the notion of algebraic degree from polynomial optimization theory, we bounded the number of critical points of the reward function over the polytope of memoryless stochastic policies.
Our analysis also yields insights into the optimization landscape, such as the number of connected components of superlevel sets of the expected reward.
Finally, we use a navigation problem in a grid world to demonstrate that the polynomial programming formulation can offer a computationally feasible approach to the reward maximization problem.
Our analysis focuses on inﬁnite-horizon problems and memoryless policies with ﬁnite state, observation, and action spaces.
Continuous spaces are interesting avenues, since they occur in real world application like robotics.
The general bound on the number of critical points in Theorem 20 does not exploit the special multi-homogeneous structure of the problem, which could allow for tighter bounds as illustrated in Proposition 21 for blind controllers.
Computing polar degrees is a challenging problem that remains to be studied using more sophisticated algebraic tools.
Possible extensions of our work include the generalization to policies with ﬁnite memories as sketched in Appendix E.1.
Further, we believe that it is interesting to explore to what extend our results can be used to identify policy classes guaranteed to contain maximizers of the reward in POMDPs.
A PPENDIX
The Sections A–D of the Appendix correspond to the Sections 2–5 of the main body.
We present the postponed proofs and elaborate various remarks in more detail.
In Appendix E we discuss possible extensions of our results to memory policies and polynomial POMDPs.
In Appendix F we provide details on the example plotted in Figure 1 and provide a plot of a three dimensional state-action frequency set.
A Details on the Preliminaries
A.1 Partially observable Markov decision processes
A.2 Semialgebraic sets and their face lattices
B Details on the Parametrizaton of Discounted State-Action Frequencies
B.1 The degree of determinantal polynomials . . .
B.2 The degree of POMDPs . . . . . . . . .
B.3 Properties of degree-one rational functions . . . . . . . . . . . . . . . . . .
C Details on the Geometry of State-Action Frequencies
C.
The fully
observable case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C
2 The partially observable case . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D Details on the Optimization
D.1 Introduction to algebraic degrees . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.2 General upper bound on the number of critical points .
D.3 Number of critical points in a two-action blind controller . . . . . . . . . . . . . .
D.4 Examples with multiple smooth and non-smooth critical points . . . . . . . . . . .
D.5 (Super)level sets of (PO)MDPs . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E Possible Extensions
E.1 Application to ﬁnite memory policies . .
E.2 Polynomial POMDPs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F Examples
F.
Toy example of Figure 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F.
Navigation in a grid world . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F.
A three dimensional example . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A D ETAILS ON THE P RELIMINARIES
We elaborate the proofs that where ommited or only sketched in the main body.
A. P ARTIALLY OBSERVABLE M ARKOV DECISION PROCESSES
The statement of Proposition 1 can found in the work by Howard (1960) and we quickly sketch the proof therein.
In order to show that the expected state-action frequencies exist without any assumptions, we recall that for a (row or column) stochastic matrix P , the Ces`aro mean is deﬁned by
[Equation]
and exists without any assumptions.
Further, P ∗ is the projection onto the subspace of stationary distribution (Doob, 1953).
For γ ∈ (0 , 1) , the matrix
[Equation]
is known as the Abel mean of P , where we used the Neumann series.
By the Tauberian theorem, it holds that P ∗ γ → P ∗ for γ (cid:37) 1 (Gillette, 1958; Hunter, 1983).
Proposition 1 (Existence of state-action frequencies and rewards) .
Let ( S , O , A , α, β, r ) be a
POMDP, γ ∈ (0 , 1] and µ ∈ ∆ S .
Then η π,µγ , ρ π,µγ and R µγ ( π ) exist for every π ∈ ∆ OA and µ ∈ ∆ S and are continuous in γ ∈ (0 , 1] for ﬁxed π and µ .
Proof.
The existence of the state-action frequencies as well as the continuity with respect to the discount parameter follows directly from the general theory since
[Equation]
for γ ∈ (0 , 1) and η π,µ= ( P Tπ ) ∗ ( µ ∗ ( π ◦ β )) .
With an analogue argument, the statement follows for the state frequencies and for the reward.
For γ = 1 we work under the following standard assumption in the (PO)MDP literature 5 .
Assumption 2 (Uniqueness of stationary disitributions) .
If γ = 1 , we assume that for any policy π ∈ ∆ OA there exists a unique stationary distribution η ∈ ∆ S×A of P π .
The following proposition shows in particular that for any initial distribution µ , the inﬁnite time horizon state-action frequency η π,µis the unique stationary distribution of P π .
Proposition 3 (State-action frequencies are discounted stationary) .
Let ( S , O , A , α, β, r ) be a
POMDP, γ ∈ (0 , 1] and µ ∈ ∆ S .
Then η π,µγ is the unique element in ∆ S×A satisfying
[Equation]
Further, ρ π,µγ is the unique element in ∆ S satisfying ρ π,µγ
[Equation]
Proof.
By the general theory of Ces`aro means, ( P Tπ ) ∗ projects onto the space of stationary distributions and hence the η π,µ
[Equation]
∗
[Equation]
π ◦ β )) is stationary.
Hence, by Assumption 2, η π,µis the unique stationary distribution.
For γ ∈ (0 , 1) we have
[Equation]
which yields the claim.
For the state distributions ρ π,µ γ the claim follows analogously or by marginalisation.
A. S EMIALGEBRAIC SETS AND THEIR FACE LATTICES
We recall the deﬁnition of semialgebraic sets, which are fundamental objects in real algebraic geometry (Bochnak et al., 2013).
A basic (closed) semialgebraic set A is a subset of R d deﬁned by ﬁnitely many polynomial inequalities as
[Equation]
where I is a ﬁnite index set and the p i are polynomials.
A semialgebraic set is a ﬁnite union of basic (not necessarily closed) semialgebraic sets, and a function is called semialgebraic if its graph is semialgebraic.
By the Tarski-Seidenberg theorem the range of a semialgebraic function is semialgebraic.
A simple algebraic set has a lattice associated to it, induced by the set of active inequalities.
More precisely, for a subset J ⊆
I we set F J :=
{ x ∈ A | p j ( x ) = 0 for j ∈ J } and endow the set F := { F J | J ⊆
I } with the partial order of inclusion.
We call the elements of F the faces of A .
The faces described above are a generalization of the faces of a classical polytope and a special case of the faces of an abstract polytope and we refer to McMullen & Schulte (2002) for more details.
Next, we want to endow this partially ordered set with more structure.
A lattice F carries two operations, the join ∧ and the meet ∨ , which satisfy the absortion laws F ∨ ( F ∧ G )
= F and F ∧ ( F ∨ G ) = F for all F, G ∈ F .
In the face lattice of a basic semialgebraic set, the join and meet are given by
[Equation]
A morphism between two lattices F and G is a mapping ϕ : F → G that respects the join and the meet, i.e., such that ϕ ( F ∧ G ) =
ϕ ( F ) ∧ ϕ ( G ) and ϕ ( F ∨ G ) =
ϕ ( F ) ∨ ϕ ( G ) for all F, G ∈ F .
A lattice isomorphism is a bijective lattice morphism where the inverse is also a morphism.
We say that two basic semialgebraic sets with isomorphic face lattice are combinatorially equivalent .
B D ETAILS ON THE P ARAMETRIZATON OF D ISCOUNTED S TATE -A CTION F REQUENCIES B. T HE DEGREE OF DETERMINANTAL POLYNOMIALS
Determinantal representation of polynomials play an important role in convex geometry (see for example Helton & Vinnikov, 2007; Netzer & Thom, 2012), but often the emphasis is put on symmetric matrices.
We adapt those arguments to the general case and present them here.
We call p a determinantal polynomial if it admits a representation
[Equation]
for all x ∈ R m , (
[Equation]
for some A 0 , . . .
, A m ∈ R n × n .
Let us use the notations
A ( x )
[Equation]
(cid:88) i = x
[Equation]
A
[Equation]
and B ( x )
[Equation]
=
[Equation]
(cid:88) i
= x i A i .
Proposition 22 (Degree of monic univariate determinantal polynomials) .
Let A, B ∈ R n × n and A be invertible and let λ 1 , . . .
, λ n ∈ C denote the eigenvalues of A − 1 B if repeated according to their algebraic multiplicity.
Then,
[Equation]
) is a polynomial of degree
[Equation]
The roots of p are given by {− λ
| j ∈ J }
Proof.
Let J ⊆ { 1 , . .
. , n } denote the set of indices j such that λ j (cid:54) = 0 .
For x (cid:54) =
we have
[Equation]
which is a polynomial of degree | J | .
Note that | J | is upper bounded by the complex rank of A − 1 B .
Since the rank over C and R agree for a real matrix, we have deg( p ) ≤ rank( A − 1 B )
= rank( B ) .
Assume now that A − 1 B is symmetric, then the rank of A − 1 B coincides with the number | J | of non zero eigenvalues.
Further, the rank of B and A − 1 B is the same.
Remark 23.
Note that the degree of p can be lower than rank( B ) , for example if
A =
I and B
=
[Equation]
Then we have rank( B ) = 1 , but p
[Equation]
=
[Equation]
=and therefore deg( p ) = 0 .
Note that in this case A − 1 B = B has no non-zero eigenvalues.
Now we show that the degree of p is still bounded by rank( B ) even if A is not invertible.
However, we loose an explicit description of the degree in this case.
Proposition 24 (Degree of univariate determinantal polynomials) .
Let A, B ∈ R n × n and consider the polynomial p : R → R ,
[Equation]
A
[Equation]
tB ) .
Then either p = 0 or if p ( t 0 ) (cid:54) = 0 and λ 1 , . .
. , λ n ∈ C denote the eigenvalues of ( A + t 0 B ) − 1 B repeated according to their algebraic multiplicity, then p has degree
[Equation]
In particular, it always holds that deg( p ) ≤ rank( B ) .
Proof.
Let without loss of generality p ( t 0 ) (cid:54) = 0 , then C
=
A + t 0 B is invertible.
By Proposition 22, the degree of q ( t ) =
det( C + tB ) is precisely |{ j | λ j (cid:54)
= 0 }| .
Noting that p ( t ) =
q ( t − t 0 ) yields the claim.
The following result generalizes Proposition 24 to multivariate determinantal polynomials.
Proposition 25 (Degree of determinantal polynomials) .
Let p : R m → R be a determinantal polynomial with the representation (7).
Then
[Equation]
Proof.
Let us ﬁx x ∈ R m and for t ∈ R set f ( t ) :
= p ( tx ) =
det( A 0 + tB ( x )) .
By the next proposition it sufﬁces to show that deg( f ) ≤ rank( B ( x )) .
However, this is precisely the statement of Proposition 24.
Proposition 26 (Degree of polynomials) .
Let p : R n → R be a polynomial.
Then there is a direction x ∈ R n such that t (cid:55)→ p ( tx ) is a polynomial of degree deg( p ) .
Moreover, for any x ∈ R n , the univariate polynomial t (cid:55)→ p ( tx ) has degree at most deg( p ) .
Proof.
Let without loss of generality p be non trivial.
Decompose p into its leading and lower order terms p =
p 1 + p 2 and choose x ∈ R n such that p 1 ( x ) (cid:54) = 0 .
Let k :=
deg( p ) , then we have p 1 ( tx ) =
t k p 1 ( x ) for all µ ∈ R .
Since the degree of t (cid:55)→ p 2 ( tx ) is at most k − 1 , the degree of t (cid:55)→ p ( tx )
= p 1 ( tx ) + p 2 ( tx ) is k .
Remark 27.
Analogue to the univariate case, it is possible to give a precise statement on the degree, which is the following.
If p is not vanishing and x 0 ∈ R m is such that p ( x 0 ) (cid:54) = 0 , then A ( x 0 ) ∈
R n × n is invertible.
Writing λ 1 ( x ) , . .
, λ n ( x ) ∈ C for the eigenvalues of A ( x 0 ) − 1 B ( x ) and J ( x ) for the indices j such that λ j ( x ) (cid:54) =
we obtain
[Equation]
B. T HE DEGREE OF POMDP S
The general bounds on the degree of determinantal polynomials directly implies Theorem 4, which we state again for the sake of convenience here.
Theorem 4 (Degree of POMDPs) .
Let ( S , O , A , α, β, r ) be a POMDP, µ ∈ ∆ S be an initial distribution and γ ∈ (0 , 1) a discount factor.
The state-action frequencies η π,µγ and ρ π,µγ , the value function V πγ and the expected cumulative reward R µγ ( π ) are rational functions with common denominator in the entries of the policy π .
Further, if they are restricted to the subset Π ⊆ ∆ OA of policies which agree with a ﬁxed policy π 0 on all states outside of
O ⊆ O , they have degree at most (cid:12) (cid:12) (cid:8) s ∈ S | β ( o | s ) > 0 for some o ∈ O (cid:9)(cid:12) (cid:12) .
In fact, the results from the preceding pararaph imply the following sharper versions.
Theorem 28 (Degree of discounted state-action frequencies of POMDPs) .
Let ( S , O , A , α, β, r ) be a POMDP, µ ∈ ∆ S be an initial distribution and γ ∈ (0 , 1) a discount factor.
Then the discounted state-action distributions can be expressed as
[Equation]
for every π ∈ ∆ OA and s ∈ S
[Equation]
where
[Equation]
are polynomials in the entries of the policy.
Further, if q s,a and q are restricted to the subset Π ⊆ ∆ OA of policies which agree with a ﬁxed policy π 0 on all states outside of
O ⊆ O
and if we set
S :=
{ s ∈ S | β ( o | s ) > 0 for some o ∈ O } , then they have degree at most
[Equation]
and
[Equation]
Proof.
Recall that we have
[Equation]
Further, by Proposition 3, the state distribution is given by ρ π,µγ = (1 − γ )(1 − γp Tπ ) − 1 µ .
Applying
Cramer’s rule yields
[Equation]
where ( I − γp Tπ ) µs denotes the matrix obtained by replacing the s -th row of I − γp Tπ with µ , which shows (8).
For the estimates on the degree, we note that
[Equation]
Further, we can use Proposition 25 to estimate the degree over a subset Π ⊆ ∆ SA of policies which agree with a ﬁxed policy π 0 on all states outside of O ⊆ O .
We obtain
[Equation]
which shows the ﬁrst estimate in (9).
To see the second inequality, we ﬁrst assume that s ∈ S .
Then ( p Tπ − p
Tπ 0 )
s has at most | S | − 1 non zero columns and hence rank at most | S | − 1 .
If s / ∈ S , then with the same argument, the rank of ( p Tπ − p Tπ 0 )
s = p
Tπ − p
Tπ 0 is at most | S | and the second estimate in (9) holds in both cases.
The estimates in (10) follow with completely analoguous arguments.
Remark 29.
The polynomial q is independent of the initial distribution µ , whereas the polynomials q s,a and therefore also their degrees depend on µ .
Further, Proposition 24 contains an exact expressions for the degree of the polynomials q s,a and q depending on the eigenvalues of certain matrices.
Corollary 30 (Degree of the reward and value function) .
Theorem 28 also yields the rational degree of the reward and value function.
Indeed, it holds that
[Equation]
where we used the matrix determinant lemma (Vrabel, 2016), and
[Equation]
The degree of their denominator is bounded by (10).
The degree of the numerator of V πγ ( s ) is bounded by (9).
Finally, the degree of the numerator of the reward R µγ is bounded by the maximum degree of the numerators of V πγ ( s ) over the support of µ .
An explicit formula for the rational degrees can be deduced from Remark 27.
Corollary 31 (Degree of curves) .
Let ( S , A , α, r ) be a MDP, µ ∈ ∆ S be an initial distribution, γ ∈ (0 , 1) a discount factor and r ∈ R S×A .
Further, let π 0 , π 1 ∈ ∆ SA be two policies that disagree on k states.
Let η λ and V λ denote the discounted state-action frequencies and the value function belonging to the policy π λ :=
π 0 + λ ( π 1 − π 0 ) .
Then both λ (cid:55)→ η λ and λ (cid:55)→ V λ are rational functions of degree at most k
. B. P ROPERTIES OF DEGREE - ONE RATIONAL FUNCTIONS B.3. A LINE THEOREM FOR DEGREE - ONE RATIONAL FUNCTIONS
First, we notice that certain degree-one rational functions map lines to lines which implies that they map polytopes to polytopes.
Further, the extreme points of the range lie in the image of the extreme points which implies that degree-one rational functions are maximized in extreme points – just like linear functions.
Deﬁnition 32.
We say that a function f : Ω → R m is a rational function of degree at most k with common denominator if it admits a representation of the form f i = p
i /q for polynomials p
i and q of degree at most k .
Remark 33.
We have seen that the state-action frequencies, the reward function and the value function of POMDPs are rational functions of degree at most |S| with common denominator.
In the case of MDPs and if a policy is ﬁxed on all but k states, it is a rational function with common denominator of degree at most k .
Proposition 34.
Let Ω ⊆
R
d be convex and f : Ω → R m be a rational function of degree at most one with common denominator and the representation
f i ( x ) =
p i ( x ) /q
( x ) for afﬁne linear functions p i , q .
Then, f maps lines to lines.
More precisely, if x 0 , x 1 ∈ Ω , then
[Equation]
is strictly increasing and satisﬁes
[Equation]
Further, c is strictly convex if | q ( x 1 ) | < | q ( x 0 ) | , strictly concave if | q ( x 1 ) | > | q ( x 0 ) | and linear if | q ( x 0 ) | = | q ( x 1 ) | .
Proof.
We set x λ :=
(1 − λ )
x 0 + λx 1 and by explicit computation we obtain
[Equation]
Noting that
[Equation]
and
[Equation]
yields (11).
Finally, we differentiate and obtain
[Equation]
Since q has no root in Ω it follows that q ( x 0 ) and q ( x 1 ) have the same sign and hence c (cid:48)
( λ ) > 0 .
Differentiating a second time yields
[Equation]
Using that sgn( q ( x λ ))
=
sgn
( q ( x 0 ))
=
sgn
( q ( x 1 )) yields the assertion.
Remark 35.
The formula (11) holds for all λ ∈ R for which x λ = λx
+ (1 − λ ) x 1 ∈ Ω .
Proposition 36 (Level sets of degree one rational functions) .
Let Ω ⊆
R
d be convex and f : Ω → R be a rational function of degree at most one.
Then, L α :=
{ x ∈ Ω | f ( x ) = α } is the intersection of an afﬁne space with Ω .
Proof.
For x, y ∈ L α the ray { x + t
( y − x ) | t ∈ R }∩ Ω is contained in L α by the line theorem.
B.3. E XTREME POINTS OF DEGREE - ONE RATIONAL FUNCTIONS
It is well known that linear functions obtain their maxima on extreme points.
We show that this is also the case for rational functions of degree at most one.
Deﬁnition 37.
Let Ω ⊆ R d .
Then we call x ∈ Ω an extreme point of Ω if x is not the strict convex combination of two other points in Ω ,
i.e., if x = (1 − λ ) x 0 + λx 1 for x 0 , x 1 ∈ Ω and λ ∈ (0 , 1) implies x 0 = x 1 = x .
We denote the set of extreme points of Ω by extr(Ω) .
Proposition 38.
Let Ω ⊆
R
d be convex and f : Ω → R m be a rational function of degree at most one with common denominator.
Then f (Ω) is convex and we have extr( f (Ω)) ⊆
f (extr(Ω)) .
Proof.
Let y
= f
( x 0 ) , y 1 = f ( x 1 )
∈ f (Ω) .
Then by the line theorem, the line connecting y 0 and y 1 agrees with the image of the line connecting x 0 and x 1 under f , in particular, it is contained in f (Ω) which shows the convexity of f (Ω) .
Pick now an extreme point y =
f ( x ) ∈ extr( f (Ω)) .
If x ∈ extr(Ω) , there is nothing to show, so let x / ∈ extr(Ω) .
Then by the Carath ´ eodory theorem we can write x as a strict convex combination (cid:80) n i =
λ i x i , λ i > 0 , n ≥ 2 for some extreme points x i ∈ extr(Ω) .
In particular, it is possible to write x as the strict convex combination x
= (1 − λ ) x 0 + λxby setting x 0 :
= (cid:80) ni
=
λ i x i .
Now, by the line theorem we have
[Equation]
where c ( λ ) ∈ (0 , 1) .
Since y is an extreme point, this implies f ( x 0 ) =
f ( x 1 ) =
y .
In particular, this shows that y
=
f
( x 1 ) ∈
f
(extr(Ω)) .
Corollary 39 (Maximizers of degree-one rational functions) .
Let Ω ⊆
R
d be a convex and compact set and let f : Ω → R be a rational function of degree at most one with common denominator.
Then f is maximized in at least one extreme point of Ω .
In particular, if Ω is a polytope, f is maximized in at least one vertex.
Proof.
Since Ω is compact and f is continuous, f (Ω) is a compact interval, lets say f (Ω) =
[ α, β ] .
By the preceding proposition we have { α, β } = extr( f (Ω)) ⊆
f (extr(Ω)) , which shows that f is maximized in at least one extreme point.
Corollary 40.
Let P ⊆
R
d be a polytope and f : Ω → R m be a rational function of degree at most one with common denominator.
Then f ( P ) is a polytope and we have vert( f ( P )) ⊆
f (vert( P )) .
Proof.
By the preceding proposition, f ( P ) is convex.
Further, f ( P ) has ﬁnitely many extreme points since extr( f ( P )) ⊆
f (extr( P ))
=
f (vert( P )) , which implies the assertion.
Proposition 41.
Let f : P
→ R m be deﬁned on the Cartesian product P = P 1 × · · · × P k of polytopes, which is a degree-one rational function with common denominator whenever all but one components are ﬁxed.
Then f ( P ) has ﬁnitely many extreme points and it holds that
[Equation]
In particular, if m = 1 this shows that f is maximized in at least one vertex of P .
Proof.
Let now x = ( x (1) , . . .
, x ( k ) ) ∈ P 1 × · · · × P k be such that f ( x ) ∈ extr( f ( P )) .
If x ( i ) ∈ vert( P i ) , there is nothing to show.
Hence, we assume that x ( i ) / ∈ vert( P i ) .
Let us denote the restriction of f onto P i by g , where we keep the other components ﬁxed to be x ( j ) .
Then we have g ( x ( i ) )
∈ extr( g ( P i )) and hence by Proposition 38 there is ˜ x ( i ) ∈ vert( P i ) such that g (˜ x ( i ) )
=
g ( x ( i ) )
=
f
( x ) .
Replacing x
( i ) by ˜ x
( i ) and iterating over i yields the claim.
Remark 42.
We have seen that both the value function as well as the discounted state-action frequencies are degree-one rational functions in the rows of the policy in the case of full observability.
Hence, the extreme points of the set of all value functions and of the set of discounted state-action frequencies are described by the proposition above.
In fact we will see later that the discounted state-action frequencies form a polytope; further, one can show that the set of value functions is a ﬁnite union of polytopes (see Dadashi et al., 2019).
B.3. I MPLICATIONS FOR POMDP S
Proposition 6.
Let ( S , A , α, r ) be a MDP and γ ∈ (0 , 1) .
Further, let π 0 , π 1 ∈ ∆ SA be two policies that differ on at most k states.
For any λ ∈ [0 , 1] let V λ ∈ R S and η µλ ∈ ∆ S×A denote the value function and state-action frequency belonging to the policy π
+ λ ( π 1 − π 0 ) with respect to the discount factor γ , the initial distribution µ and the instantaneous reward r .
Then the rational degrees of λ (cid:55)→ V λ and λ (cid:55)→ η λ are at most k .
If they differ on at most one state ˜ s ∈ S then
V λ
= V
+ c ( λ ) · ( V 1 − V 0 ) and η µλ = η µ 0 + c ( λ ) · ( η µ 1 − η µ 0 ) for all λ ∈ [0 , 1] , where
[Equation]
Proof.
This is a direct consequence of Proposition 34 and Theorem 4.
Remark 43.
The proposition above describes the interpolation speed λ · ρ µλ (˜ s ) /ρ µ 1 (˜ s ) in terms of the discounted state distribution in ˜ s .
This expressions extends to the case of mean rewards – note that the determinants vanish – and the theorem can be shown to hold in this case as well, if we set 0 / 0 :
= 0 .
Note that the interpolation speed does not depend on the initial condition µ .
Remark 44.
Reﬁnements on the upper bound of the rational degree of λ (cid:55)→ V λ and λ (cid:55)→ η λ can be obtained using Proposition 24.
Indeed, if we write η λ ( s, a ) =
q sa ( λ ) /q
( λ ) like in Theoremthose degrees can be upper bounded by
[Equation]
where S ⊆ S is the set of states on which the two policies differ; see also the proof of Theoremfor more details on an analogue argument.
Hence, the degree of the two curves λ (cid:55)→ V λ and λ (cid:55)→ η λ is upper bounded by rank( p 1 − p 0 ) .
C D ETAILS ON THE G EOMETRY OF S TATE -A CTION F REQUENCIES
The set of all state-action frequencies is known to be a polytope in the fully observable case (Derman, 1970) and we show that it is combinatorially equivalent to the conditional probability polytope ∆ SA .
We show that in the partially observable case the set of feasible state-action frequencies is cut out from this polytope by a ﬁnite set of polynomial inequalities.
We discuss the special structure of those polynomials and give closed form expressions for them.
C. T HE FULLY OBSERVABLE CASE
Let ν π,µγ ∈ ∆ S×S denote the expected number of transitions from s to s (cid:48) given by
[Equation]
respectively.
Note that we have
[Equation]
[Equation]
hence ν π,µγ is the image of η π,µγ under the linear transformation
[Equation]
Therefore
[Equation]
we can hope to obtain a characterization of N µγ using this mapping.
In order to do so
[Equation]
we would like to understand the structural properties of ν π,µγ .
For γ = 1 those distributions have equal marginals since we can compute
[Equation] [Equation]
In the discounted case
[Equation]
we compute similarly
[Equation]
If we perceive ν π,µγ ∈ ∆ S×S as a matrix, we have shown that
[Equation]
which motivates the following deﬁnition.
We will see that the set of state-action frequencies is the pre-image of the following polytope under a linear map.
Deﬁnition 45 (Discounted Kirchhoff polytopes) .
For a distribution µ ∈ ∆ S and γ ∈ (0 , 1] we deﬁne the discounted Kirchhoff polytope (this is a generalization of a deﬁnition by Weis 2010)
[Equation]
where 1 ∈ R S is the all one vector.
So far, we have observed that f α ( η π,µγ ) ∈
Ξ µγ , i.e., that
[Equation]
maps to f − 1 α (Ξ µ γ ) .
In order to see that this mapping is surjective on f − 1 α (Ξ µ γ ) we show that its right inverse is given through conditioning.
The following proposition uses the ergodicity assumption.
Proposition 46.
Let γ ∈ (0 , 1] and η ∈ f − 1 α (Ξ µγ ) and let ρ ∈ ∆ S denote the state marginal of η .
Set
[Equation]
arbitrary element in ∆ A
if ρ ( s ) = 0 , then we have η π,µγ = η .
Proof.
We calculate
[Equation]
The unique characterization from Proposition 3 of η π,µ γ yields the assertion.
The proposition states that we can reconstruct the policy from the state-action frequencies by conditioning and is well known in the context of the dual linear programming formulation of MDPs (Kallenberg, 1994).
Hence, it will be convenient later to work under the following assumption in which ensures that policies in ∆ SA are one-to-one with state-action frequencies.
Assumption 13 (Positivity) .
Let ρ π,µγ > 0 hold entrywise for all policies π ∈ ∆ SA .
Note that this positivity assumption holds in particular, if either α > 0 and γ > 0 or γ < 1 and µ > 0 or entrywise.
Indeed, if α > 0 , then the transition kernel p π is strictly positive for any policy since
[Equation]
since ( π ◦ β )( a | s ) > 0 for some a ∈ A .
Since ρ π,µγ is discounted stationary with respect to p π (Proposition 3), it holds that
[Equation]
since ρ π,µγ ( s (cid:48) ) > 0 for some s (cid:48) ∈ S .
If µ > 0 and γ < 1 , then ρ π,µγ ( s ) ≥ (1 − γ ) µ ( s ) > 0 .
As a consequence of Proposition 46, we obtain the following characterization of N µγ .
Proposition 8 (Characterization of N µγ ) .
Let ( S , A , α, r ) be a MDP, µ ∈ ∆ S be an initial distribution and γ ∈ (0 , 1] .
It holds that
[Equation]
) where w sγ := δ s ⊗ 1 − γα ( s |· , · ) .
For γ ∈ (0 , 1) , ∆ S×A can be replaced by [0 , ∞ ) S×A in (3) .
Instead of proving this proposition directly, we ﬁrst present the following version of it.
Proposition 47.
Let ( A , S , α, r ) be an MDP and γ ∈ (0 , 1] .
It holds that N µγ = f − 1 α (Ξ µγ ) .
Proof.
By (14) and (15), it holds that f α ( N µγ ) ⊆
Ξ µγ and thus N µγ ⊆
f − 1 α (Ξ µγ ) .
However, by
Proposition46, for every η ∈ f α (Ξ µγ ) there is a policy π ∈ ∆ SA such that η π,µγ = η
and hence it holds that f − 1 α (Ξ µγ ) ⊆
f − 1 α (Ξ µγ ) .
Proof of Proposition 8.
By the preceding proposition η ∈ N µγ is equivalent to η ∈ ∆ S×A and ν :=
f α ( η ) ∈ Ξ µγ .
Using the deﬁnition of Ξ µγ this equivalent to
[Equation]
for all s ∈ S .
Plugging in the deﬁnition of f α we see that the term on the left hand side is equivalent to
[Equation]
The ﬁrst term of the right hand side is precisely γ (cid:88)
[Equation]
Hence, we have seen that f α ( η ) ∈
Ξ µγ is equivalent to the condition
[Equation]
all s ∈ S
[Equation]
Note that { η | (cid:104) w sγ , η (cid:105) S×A =
[Equation]
1 − γ ) µ ( s ) for all s ∈ S} =
η 0 + { w sγ | s ∈ S} ⊥ for an arbitrary element η 0 satisfying (16).
This shows the ﬁrst equation in (3).
The second equation follows from the observation that (cid:80) s w sγ
= (1 − γ ) 1 .
Hence, for γ < 1 it holds that
[Equation]
C.1. D ERIVATIVE OF THE DISCOUNTED STATE - ACTION FREQUENCIES
In this section we discuss the Jacobian of the parametrization π
(cid:55)→ η π of the discounted state-action frequencies.
One motivation for this is that this Jacobian plays an important role in the relation of critical points in the policy space and the space of discounted state-action frequencies.
Note that Ψ µγ ( π ) =
(1 − γ )(1 − γP Tπ ) − 1 ( µ ∗ π ) is well deﬁned, whenever (cid:107) P π (cid:107) 2 < γ − 1 .
Hence, we can extend Ψ µγ onto the neighborhood (cid:8) π ∈ R S×A | (cid:107) P π (cid:107) 2 < γ − 1 (cid:9) of ∆ SA , which enables us to compute the
Jacobian of Ψ µγ .
Proposition 48 (Jacobian of Ψ µγ ) .
For any policy π ∈ ∆ SA and s ∈ S , a ∈ A it holds that
[Equation]
) where ( ρ
[Equation]
[Equation]
Hence, ∂ ( s,a ) Ψ µγ ( π ) is identical to the ( s, a ) -th column of ( I − γP Tπ ) − 1 up to the scaling factor of ρ π,µγ ( s ) .
In particular, if ρ π,µγ ( s ) > 0 for all s ∈ S , the Jacobian D Ψ µγ has full rank.
Proof.
Recall that for invertible matrices
A ( t ) , it holds that ∂ t A ( t ) − 1 = − A ( t ) − 1 ( ∂ t A ( t ))
A ( t ) − 1 .
We compute
[Equation]
Further, direct computation shows
[Equation]
Using the fact that ρ π,µγ is the discounted stationary distribution
[Equation]
yields
[Equation]
which shows (17).
Note that ( I − γP Tπ ) − 1 ( δ s ⊗ δ a ) is precisely the ( s 0 , a 0 ) -th column of the matrix ( I − γP Tπ ) − 1 .
Those columns are linearly independent, and so are the partial derivatives ∂ ( s,a ) Ψ µγ ( π ) , given that the discounted stationary distribution ρ π,µγ vanishes nowhere.
Corollary 49 (Dimension of N µγ ) .
Assume that ρ π,µγ > 0 entrywise for some policy π ∈ int(∆ SA ) .
Then we have
[Equation]
Proof.
By Proposition 48 the mapping Ψ µγ is full rank in a neighborhood of π and hence, we have
[Equation]
Let us consider a parametrized policy model Π Θ =
{ π θ | θ ∈ Θ } with differentiable parametrization θ (cid:55)→ π θ .
Proposition 50 (Parameter derivatives of discounted state-action frequencies) .
It holds that ∂ θ i η π θ ,µ γ =
( I − γP T π θ ) − 1 ( ρ π θ ,µ γ ∗ ∂
θ
i π θ ) , where ( ρ π θ ,µ γ ∗ ∂ θ i π θ )( s, a ) =
ρ π θ ,µ γ ( s ) ∂ θ
i π θ ( a | s ) .
Proof.
This follows directly from the application of the chain rule and (17).
Using this expression, we can compute the parameter gradient with respect to the discounted reward
F ( θ ) :
= R µγ ( π θ ) and recover the well known policy gradient theorem, see Sutton et al.
Deﬁnition 51 (state-action value function) .
We call Q π :=
( I − γP π ) − 1 r ∈ R S×A
the state-action value function or the Q-value function of the policy π .
Corollary 52 (Policy gradient theorem) .
It holds that
[Equation]
Proof.
Using the preceding proposition
[Equation]
we compute
[Equation]
Remark 53 (POMDPs as parametrized policy models) .
The case of partial observability can sometimes be regarded as a special case of parametrized policies.
In fact the observation mechanism β induces a linear map π (cid:55)→ π ◦ β .
This interpretation together with the preceding proposition can be used to calculate policy gradients in partially observable systems
. C.1. T HE FACE LATTICE IN THE FULLY OBSERVABLE CASE
So far, we have seen that the set of state-action frequencies form a polytope in the fully observable case.
However, not all polytopes are equally complex and thus we aim to describe the face lattice of
N µγ , which describes the combinatorial properties of a polytope, see Ziegler (2012).
Theorem 54 (Combinatorial equivalence of N µγ and ∆ SA ) .
Let ( A , S , α, r ) be an MDP and γ ∈ (0 , 1] .
Then π (cid:55)→ η π,µγ induces an order preserving surjective morphism between the face lattices of ∆ SA and N µγ , such that for every
I ⊆ S ×
A it holds that
[Equation]
If additionally Assumption 13 holds, this is an isomorphism and preserves the dimension of the faces.
Proof.
First, we note that the faces of both ∆ SA and N µγ have the structure of the left and right hand side of (54) respectively, which follows from (3).
Denote now the left and right hand side in (54) by
F and G respectively, then we need to show that Ψ µγ ( F ) = G .
For π ∈ F it holds that
[Equation]
all
[Equation]
I and hence η π,µγ ∈ G .
On the other hand for η ∈ G we can set π ( ·| s ) :
= η ( ·| s ) whenever deﬁned and any other element such that π ( a | s ) = 0 for all ( s, a ) ∈
I otherwise.
Then we surely have π ∈ F and by Proposition 46 also η π,µγ = η .
In the case that ρ π,µγ > 0 holds entrywise for all policies π ∈ int(∆ SA ) , the mapping η
(cid:55)→ η ( ·|· ) deﬁnes an inverse to Ψ µγ , which shows that the mapping deﬁned in (54) is injective.
The assertion on the dimension follows from basic dimension counting, from the fact that the rank is preserved by a lattice isomorphism or by virtue of Proposition 48.
C. T HE PARTIALLY OBSERVABLE CASE
In Corollary 5, we have seen that the discounted state-action frequencies form a semialgebraic set.
Now we aim to describe its deﬁning polynomial inequalities.
In Section 5 we will discuss how the degree of these polynomials allows us to upper bound the number of critical points of the optimization problem.
Deﬁnition 9 (Effective policy polytope) .
We call the set of effective policies τ
= π ◦ β ∈ ∆ SA the effective policy polytope and denote it by ∆ S ,β A .
Note that ∆ S ,β A is indeed a polytope since it is the image of the polytope ∆ OA under the linear mapping
π (cid:55)→ π ◦ β .
Hence, we can write it as an intersection
[Equation]
18) where U , C ⊆ R S×A are an afﬁne subspace and a polyhedral cone and describe a ﬁnite set of linear equalities and a ﬁnite set of linear inequalities respectively.
In Section 4 we will compute those sets explicitely under mild conditions and see that they do not carry an afﬁne part.
C.2. D EFINING LINEAR INEQUALITIES OF THE EFFECTIVE POLICY POLYTOPE
Obtaining inequality descriptions of the images of polytopes under linear maps is a fundamental problem that is non-trivial in general.
It can be approached algorithmically, e.g., by Fourier-Motzkin elimination, block elimination, vertex approaches, and equality set projection (Jones et al., 2004).
We discuss the special case where the linear map is injective, corresponding to the case where the associated matrix B has linearly independent columns.
As a polytope is a ﬁnite intersection of closed half spaces H + = { x | n T x ≥ α } , it sufﬁces to characterize the image BH + .
It holds that
[Equation]
where B
+ is a pseudoinverse and where we have used that B + y consists of at most one element by the injectivity of B .
Let us now come back to the mapping
π (cid:55)→ π ◦ β
= βπ .
By the “vec-trick”, this map corresponds to vec( βπI ) =
( I T ⊗ β ) vec( π ) .
Hence the linear map is represented by the matrix B =
I ⊗ β .
We observe that ( I ⊗ β )
+
= I ⊗ β
+
(see Langville & Stewart, 2004, Section 2.6.3).
Notice that B
= I ⊗ β has linearly independent columns if and only if β does.
By the above discussion, if β has linearly independent columns, then an inequality (cid:104) π, n (cid:105) ≥ 0 in the policy polytope ∆ OA corresponds to an inequality (cid:104) τ, ( β + ) T n (cid:105) ≥ 0 in the polytope ∆ SA .
Assumption 10.
The matrix β ∈ ∆ SO ⊆
R S×O has linearly independent columns.
Remark 11.
The assumption above does not imply that the system is fully observable.
Recall that if β has linearly independent columns, the Moore-Penrose takes the form β + = ( β T β ) − 1 β T .
An interesting special case is when β is deterministic but may map several states to the same observation (this is the partially observed setting considered in numerous works).
In this case, β + = diag( n − 1 1 , . .
n − 1 |O| )
β T , where n o denotes the number of states with observation o .
In this case, β + so agrees with the conditional distribution
β
( s | o ) with respect to a uniform prior over the states; however, this is not in general the case since β + can have negative entries.
Theorem 12 ( H -description of the effective policy polytope) .
Let ( S , O , A , α, β, r ) be a POMDP and let Assumption 10 hold.
Then it holds that
[Equation]
where U = { π ◦ β | π ∈ R S×O } is a subspace,
C = { τ ∈ R S×A | β + τ ≥ 0 } is a pointed polyhedral cone and D = { τ ∈ R S×A | (cid:80) a ( β + τ ) oa = 1 for all o ∈ O} an afﬁne subspace.
Further, the face lattices of ∆ OA and ∆ S ,β A are isomorphic.
Proof.
First, we recall the deﬁning linear (in)equalities of the policy polytope ∆ OA , which are given by
[Equation]
and
[Equation]
Hence, by the general discussion from above, namely by (19), it holds that
[Equation]
Note that the linear inequalities (cid:80) a ( β + τ ) oa = 1 are redundant in ∆ SA .
To see this, we note that β +
=
1 by the injectivity of β and β 1 = 1 .
Now we can check that
[Equation]
This together with β (∆ OA ) ⊆
∆ SA shows that
[Equation]
The reformulation of the sets C and D for deterministic observation mechanisms β follows from the preceding remark.
C.2. D EFINING POLYNOMIAL INEQUALITIES OF THE FEASIBLE STATE - ACTION FREQUENCIES
Using that the inverse of π (cid:55)→ η π is given through conditioning (see Proposition 46), we can translate linear inequalities in ∆ SA into polynomial inequalities in N µγ .
More precisely, we have the following result, which can easily be extended to more general inequalities.
Proposition 14 (Correspondence of inequalities) .
Let ( S , A , α, r ) be a MDP, τ ∈ ∆ SA and let η ∈ ∆ S×A denote its corresponding discounted state-action frequency for some µ ∈ ∆ S and γ ∈ (0 , 1] .
Let c ∈ R , b ∈ R S×A and set S :
= { s ∈ S | b sa (cid:54) =
for some a ∈ A} .
Then (cid:88) s,a b sa τ sa ≥ c implies (cid:88)
[Equation]
where the right is a multi-homogeneous polynomial 8 in the blocks ( η sa ) a ∈A ∈ R A with multi-degree
S ∈ N S .
If further Assumption 13 holds, the inverse implication also holds.
Proof.
Let τ ∈ ∆ SA and let η denote its corresponding discounted stationary distribution and ρ the state marginal.
Assuming that the left hand side holds, we compute
[Equation]
which shows the ﬁrst implication.
If further Assumption 13 holds, the product over the marginals is strictly positive, which shows the other implication.
Remark 55.
According to the preceding proposition, a linear inequality in the state policy polytope ∆ SA involving actions of k different states yields a polynomial inequality of degree k in the set of state-action frequencies N µγ .
In particular, for a linearly constrained policy model Π ⊆ ∆ SA , where every constraint only addresses a single state, the set of state-action frequencies induced by these policies will still form a polytope.
This shows that this type of box constraints are well aligned with the algebraic geometric structure of the problem.
The linear constraints arising from partial observability never exhibit this box type structure – unless the system is equivalent to its fully observable version.
This is because the projection of the effective policy polytope ∆ S ,β A onto a single state always gives the entire probability simplex ∆ A , which is never the case, if there is a non trivial linear constraint concerning only this state.
Theorem 16.
Let ( S , O , A , α, β, r ) be a POMDP, µ ∈ ∆ S and γ ∈ (0 , 1] and assume that Assumption 13 holds.
Then we have N µ,βγ =
N µγ ∩ V ∩ B , where V is a variety described by multi-homogeneous polynomial equations and B is a basic semialgebraic set described by multihomogeneous polynomial inequalities.
Further, the face lattices of ∆ S ,β A and N µ,βγ are isomorphic.
Proof.
The equation N µ,βγ =
N µγ ∩V
∩B is a direct consequence of (4) and Proposition 14.
Further, it is clear from Proposition 14 that the mapping Ψ: ∆ SA → N µγ , π (cid:55)→ η π,µγ induces a bijection of the face lattices of ∆ S ,β A and N µ,βγ .
In order to see that the join and meet are respected, we note that for
F, G ∈ F (∆ S ,β A ) it holds that Ψ( F ∧ G ) =
Ψ( F ∩ G ) =
Ψ( F ) ∩ Ψ( G ) =
Ψ( F ) ∧ Ψ( G ) .
Further, Ψ( F ∨ G ) is a face of N µ,βγ containing Ψ( F ) and Ψ( G ) and hence by deﬁnition Ψ( F ) ∨ Ψ( G ) ⊆
Ψ( F ∨ G ) .
Further, for any face I of N µ,βγ containing Ψ( F ) and Ψ( G ) it holds that Ψ − 1 ( I ) is a face of ∆ S ,β A containing F and G and hence Ψ − 1 ( I ) ⊇
F ∨ G or equivalently Ψ( F ∨ G ) ⊆
I .
Remark 18.
By Theorem 12 and Proposition 14, the deﬁning polynomials of the basic semialgebraic set B from Theorem 16 are indexed by a ∈ A , o ∈ O and are given by p
[Equation]
where S o := { s ∈ S | β + os (cid:54) = 0 } .
The polynomials depend only on β and not on γ , µ nor α , and have | S o ||A|
|S
o |− 1 monomials of degree | S
o | of the form (cid:81) s
∈ S
o η sf ( s ) for some f : S o → A .
In particular, we can read of the multi-degree of p ao with respect to the blocks ( η sa ) a ∈A which is given by 1 S o (see also Proposition 14).
A complete description of the set N µ,βγ via (in)equalities follows from the description of N µγ via linear (in)equalities given in (3).
In Section 5 we discuss how the degree of these polynomials controls the complexity of the optimization problem.
Remark 56 (Deﬁning polynomial equalities) .
Analogously to the deﬁning inequalities, we can compute the deﬁning polynomial equalities in the following way.
First, we need to compute a basis { b j } j ∈ J of { βπ | π ∈ R O×A } ⊥ ⊆ R S×A , which can easily be done using the Gram-Schmidt process.
Note that the deﬁning linear equalities of the effective policy polytope (in the policy polytope) are given by (cid:104)
b j , τ (cid:105) S×A = 0 .
Hence, by Proposition 14 the corresponding polynomial equality is given by
[Equation]
where S j := { s ∈ S | b jsa (cid:54) = 0 for some a ∈ A} .
Remark 57.
In the fully observable case we have | S o | = 1 for each o .
Hence, each of the polynomial inequalities has a single term of degree 1 .
Indeed, in this case the inequalities are simply η sa ≥ 0 , for each a , for each s .
In the case of a deterministic β , we have β + os = 1 s ∈ S o / | S
o | .
For each o, a , there is an inequality (cid:80) f : S o →A
| f − 1 ( a ) | (cid:81) s ∈ S o η sf ( s ) ≥ 0 of degree | S
o | equal to the number of states that are compatible with o .
Remark 58 (Reformulation of reward maximization as a polynomial program) .
By the theorem above and Proposition 14, reward maximization is equivalent to the maximization of a linear function subject to polynomial constraints.
This enables the use of any (approximate) solution technique of polynomial optimization problems in order to solve POMDPs.
Such methods have been developed for a long time and have been applied to a variety of problems (Anjos & Lasserre, 2011; Lasserre, 2015).
As meta algorithm, this is presented in Algorithm 1.
Once, a solution η ∗ is obtained, the corresponding state policy τ ∗ ∈ ∆ SA can be computed by conditioning, i.e. τ ( a | s ) :
= η sa / ( (cid:80) a (cid:48)
η sa (cid:48) ) .
Then, every π ∗ ∈ ∆ OA with βπ ∗ = τ ∗ is an optimal policy.
Such a policy can be computed by solving a system of linear equations, which are βπ
= τ and π ∈ ∆ OA , which is standard.
In particular, if β has linearly independent columns, it holds that π ∗ :=
β + τ .
We demonstrate that this offers a computationally feasible approach to planning of POMPDs in Section F on the toy example used for
Figure 1 and a grid world.
Remark 59 (Polynomial constraints for deterministic observations) .
In the case, where β corresponds to a determinstic mapping we can compute all polynomial constraints in closed form.
Let us
Algorithm
Require: α, β, γ, µ for s ∈ S do w s ← δ s ⊗ 1 A − γα ( s |· , · ) end for for a ∈ A , o ∈
O do
Deﬁne p ao according to Equation (5) end for
Compute a basis { b j } j ∈ J of { βπ | π ∈ R O×A } ⊥ ⊆ R S×A for j ∈ J do
Deﬁne q j according to Equation (20) end for η ∗ ← arg max (cid:104) r, η (cid:105) sbj to η ≥ 0 , (cid:104) w s , η (cid:105) =
(1 − γ ) µ s , (cid:104) 1 S×A , η (cid:105)
= 1 , p ao ( η ) ≥ 0 , q j ( η ) =
R ∗ ← (cid:104) r, η ∗ (cid:105) τ ∗ ← η ∗ ( ·|
· ) ∈
∆ SA π ∗ ← solution of βπ = τ ∗ return maximizer η ∗ , optimal value R ∗ , optimal policy π ∗ assume that β ( o | s ) =
δ og ( s ) for some mapping g : S → O and write S o := g − 1 ( { o } ) ⊆
S , then τ ∈ ∆ SA belongs to the effective policy polytope ∆ S ,β A if and only if τ ( a | s 1 )
=
τ ( a | s 2 ) for all s 1 , s 2 ∈ S o , a ∈ A , o ∈ O .
Note that this can be encoded in (cid:80) o |A| ( | S o | − 1) =
|A| ( |S| − |O| ) linear equations; indeed if we ﬁx s o ∈ S
o , then equation 21 is equivalent to τ ( a | s ) − τ ( a | s o )
=
for all s ∈ S o \ { s o } , a ∈ A , o ∈ O .
By Proposition 14 for η ∈ N µγ it is equivalent to lie in N µ,βγ or to satisfy η sa (cid:88)
[Equation]
Note that in this case, there are no polynomial inequalities; this can also be seen from Remarkand Remark 18.
Indeed, it holds that β + = β T diag( n 1 , . . . , n |O| ) ≥ 0 , where n o :=
| S o | .
Hence, the polynomial inequalities p ao ( η ) ≥ 0 are redundant on the cone [0 , ∞ ) S×A .
D D ETAILS ON THE O PTIMIZATION
Let us quickly recall how we can reformulate the reward maximization problem as a polynomial optimization problem, which then leads us to the mighty tools of algebraic degrees.
We perceive the reward maximization problem again as the maximization of a linear function p 0 over the set of feasible state-action frequencies N µ,βγ .
Since under Assumption 13 the parametrization π (cid:55)→ η π is injective and has a full-rank Jacobian everywhere (see Appendix C.1.1), the critical points in the policy polytope ∆ OA correspond to the critical points of p 0 on N µ,βγ (Trager et al., 2019).
In general, critical points of this linear function can occur on every face of the semialgebraic set N µ,βγ .
The optimization problem thus has a combinatorial and a geometric component, corresponding to the number of faces of each dimension and the number of critical points in the interior of any given face.
We have discussed the combinatorial part in Theorem 16 and focus now on the geometric part.
Writing N µ,βγ = { η ∈ R S×A : p i ( η ) ≤ 0 , i ∈ I } , we are interested in the number of critical points on the interior of a face
[Equation]
Note that a point η ∈ int( F J ) is critical, if and only if it is a critical point on the variety
V J : = { η ∈ R S×A | p j ( η ) = 0 for j ∈ J } .
For the sake of notation, let us assume that J = { 1 , . . .
, m } from now on.
We can upper bound the number of critical points in the interior of the face by the number of critical points of the polynomial optimization problem maximize p 0 ( η ) subject to p j ( η ) = 0 for j = 1 , . . .
, m, (24) where the polynomials have n variables.
The number of critical points of this problems is upper bounded by the algebraic degree of the problem as we discuss now.
D. I NTRODUCTION TO ALGEBRAIC DEGREES
We try to present the results from the mighty theory of algebraic degrees that we use here and refer the interested reader to the excellent low level introduction by Breiding et al.
(2021) and to the references therein.
Let us consider the polynomial optimization problem (24), where we do not require p 0 to be linear.
Further, denote the number of variables by n (in the case of state-action frequencies n = |S||A| ) and denote the degrees of p 0 , . .
, p m by d 0 , . . .
, d m .
We call a point critical , if it satisﬁes the KKT conditions ( ∇ p 0 ( x ) + (cid:80) mi
=1 λ i ∇ p
i ( x ) = 0 , p 1 ( x ) = · · · = p m ( x ) = 0 ), which can be phrased as a system of polynomial equations (see Nie & Ranestad, 2009).
The number of complex solutions to those criticality equations, when ﬁnite, is called the algebraic degree of the problem.
The algebraic degree is determined by the nature of the polynomials p 0 , . . .
, p m and captures the computational complexity of the optimization problem (Kung, 1973; Bajaj, 1988).
A special case of (24) is when m = n and the polynomials p 1 , . .
. , p m are generic.
Then by
B´ezout’s theorem there are exactly d 1 · · · d n isolated points satisfying the polynomial constraints and all of them are critical and hence the algebraic degree is precisely d 1 · · · d n (Timme, 2021).
If the polynomials p 0 , . .
. , p m deﬁne a complete intersection, i.e., the co-dimension of their induced variety is m + 1 , the algebraic degree of (24) is upper bounded by
[Equation]
and this bound is attained for generic polynomials (Nie & Ranestad, 2009; Breiding et al., 2021).
For non-complete intersections, the expression (25) does not need to yield an upper bound if some constraints are redundant.
However, we can modify the expression to obtain a valid upper bound.
Indeed, if l and c = n − l denote the dimension and co-dimension of
[Equation]
and if p 0 is generic and if the degrees are ordered, i.e., d 1 ≥ · · · ≥ d m , then the algebraic degree is upper bounded by
[Equation]
To see this, ﬁx a subset J ⊆ { 1 , . .
, m } of cardinality c , such that V
= { x | p j ( x ) = 0 for j ∈ J } .
Then we can apply the bound from (25) and evaluate it to be
[Equation]
which is clearly upper bounded by (26)
If p 0 is linear, then d 0 = 1 and the expression simpliﬁes to
[Equation]
If further d i = 1 for i ≥ k for some k ≤ c , then we obtain
[Equation]
If p k +1 , . .
. , p m are afﬁne linear (and in general position relative to p 1 , . .
. , p k , the algebraic degree of (24) is given by the ( m − k ) -th polar degree δ m − k
( V ) of the variety
[Equation]
see Draisma et al.
(2016); ¨Ozl¨um C¸elik et al.
This relation is particularly useful, since for state-action frequencies there are always active linear equations as described in (3).
The polar degrees of certain interesting cases (Segre-Veronese varieties) have been recently computed by
Sodomaco (2020, Section 5) and our proof of Proposition 21 builds on those formulas and their presentation by ¨Ozl¨um C¸elik et al.
Remark 60 (Genericity assumptions) .
In the case, where the polynomials p 0 , . . .
, p m are not generic, there might be inﬁnitely many critical points.
Indeed, even for a linear program, i.e., when all polynomials are linear, there might be inﬁnitely many and even a non-trivial face of global optima.
This is however not the case if p 0 is generic.
Hence, the genericity assumptions on the reward vector r and also other elements of the POMDP are not surprising.
For example, they prevent the reward vector to be identical to zero or to be perpendicular on all vectors δ s ⊗ 1 − γα ( s |· , · ) in which cases the reward function would be constant and every policy would be a global optimum.
D. G ENERAL UPPER BOUND ON THE NUMBER OF CRITICAL POINTS
Theorem 20.
Consider a POMDP ( S , O , A , α, β, r ) , γ ∈ (0 , 1) , assume that r is generic, that β ∈ R S×O is invertible, and that Assumption 13 holds.
For any given I ⊆
A × O consider the following set of policies, which is the relative interior of a face of the policy polytope:
[Equation]
Let O :=
{ o ∈ O | ( a, o ) ∈
I for some a } and set k
o := |{ a | ( a, o ) ∈ I }| as well as d o := |{ s | β + os (cid:54) = 0 }| .
Then, the number of critical points of the reward function on int( F ) is at most
[Equation]
where l = |S| ( |A| − 1) − | I | .
If α and µ are generic, this bound can be reﬁned by computing the polar degrees of multi-homogeneous varieties (see Proposition 21 for a special case).
The same bound holds in the mean reward case γ = 1 for l given in Remark 61.
Proof.
The face G of the effective policy polytope corresponding to F is given by
[Equation]
In order to describe the corresponding set of discounted state-action frequencies, we use the notation
[Equation]
then it holds that
[Equation]
Then, F and G correspond to the face
[Equation]
and equality if and only if ( a, o
[Equation]
In order to use the explicit description of N µγ given in (3), we remind the reader that w sγ := δ s ⊗ 1 − γα ( s |· , · ) .
Then, it holds that int
[Equation]
η ∈ [0 , ∞ ) S×A | p ao ( η ) ≥ 0 and equality if and only if ( a, o ) ∈
I
[Equation]
for
[Equation]
where we used Proposition 8.
Since the discounted state distributions are all positive by assumption, for η ∈ int( H ) it holds η sa =
if and only if τ ( a | s ) :
= η ( a | s ) = 0 .
Note that τ
= π ◦ β for some π ∈ ∆ OA by assumption and thus for η ∈ int( H ) it holds that η sa =
if and only if
[Equation]
which holds if and only if ( a, o ) ∈ I for every o ∈ O with β ( o | s ) > 0 .
Hence, if we write
J :=
{ ( s, a ) | ( a, o ) ∈
I for all o ∈ O with β ( o | s ) > 0 } , we obtain int( H ) =
(cid:110) η ∈ R S×A | η sa ≥ 0 and equality if and only if ( s, a ) ∈ J, η T w sγ
= (1 − γ ) µ s for s ∈ S , p ao ( η ) ≥ 0 and equality if and only if ( a, o ) ∈
I (cid:111) .
The number of critical points over this surface is upper bounded by the number of critical points over
[Equation]
Now we want to apply (27) and note that the objective p
= r is generic.
Further, we see that there are | I | non-linear constraints and hence in the notation of (27) have k = | I | .
Further, we can calculate to dimension and co-dimension of V as follows.
Note that F → V , π (cid:55)→ η π is a local parametrization of V (meaning it parametrizes a full dimensional subset of V ), which is injective and has full rank Jacobian everywhere.
Hence, we have
[Equation]
The co-dimension of V is given by |S||A|− dim( V ) =
|S| + | I | and with the notation from (27), we have c = |S| + | I | ≥ k .
Further, it holds that deg( p ao ) ≤ d o and using (27) yields an upper bound of
[Equation]
Remark 61 (The mean reward case) .
Theorem 20 can be generalized to the mean reward case, i.e., to the case of γ = 1 with some adjustments.
Indeed, the proof can be carried out analogously, however, the characterization of N µ 1 has the extra linear condition that (cid:80) sa η sa = 1 , see also
Proposition 8.
Indeed, in the mean reward case we have with the notation from the proof above int( H ) =
(cid:110) η ∈ R S×A | η sa ≥ 0 and equality if and only if ( s, a ) ∈ J, η T w sγ = 0 for s ∈ S , (cid:88)
sa η sa = 1 , p ao ( η ) ≥ 0 and equality if and only if ( a, o ) ∈
I (cid:111) .
Hence, the upper bound in (6) remains valid if we set
[Equation]
In the discounted case we obtained an explicit formulation for l .
In the mean case the value obeys a case distinction depending, in particular, on whether the all ones vector 1 lies in the span of the vectors w sγ .
However, the value can be computed from the above expression equation 28 in any given speciﬁc case.
Corollary 62 (Critical points of MDPs) .
Consider a MDP ( S , A , α, r ) , γ ∈ (0 , 1) , assume that r is generic, that β ∈ R S×O is invertible, and that Assumption 13 holds.
Then, every critical point π ∈ ∆ O A of the discounted expected reward function is deterministic.
Proof.
We evaluate the bound of Equation (6).
If the face is not a vertex, then the corresponding index set I ⊆
A × O satisﬁes | I | < |O| ( |A| − 1) and thus in the notation from Theorem 20 it holds that l > 0 .
Note that d o = 1 for every o ∈ O and hence there is at least one factor in the product in (6) that vanishes and so does the whole expression in (6).
Remark 63 (Geometry around the critical points) .
The key argument in the proof of Theoremis that a critical point π ∈ ∆ OA of the reward function corresponds to a critical point η of a linear function over a multi-homogeneous variety V , where the deﬁning polynomials can be computed by the means of Proposition 8 and Proposition 14.
A closer study of this variety would shed light into the geometry of the loss landscape around the critical points, which has important implications for gradient based methods.
Remark 64 (Efﬁcient design of observation mechanisms) .
The bound (6) could be used to design observation mechanisms in such a way that the reward function has the least critical points, which would potentially make the system more approachable for gradient based methods.
(Rauh et al., 2021) showed that planning in POMDPs is stable under perturbations of the observation kernel β .
More precisely, consider two observation kernels β, β (cid:48) ∈
∆ SO satisfying (cid:107) β
·| s ) − β (cid:48) ( ·| s ) (cid:107) TV = (cid:80) o | β ( o | s ) − β (cid:48)
( o | s ) | / 2 ≤ ε for every s ∈ S .
Then if π ∈ ∆ OA is an optimal policy of ( S , A , O , α, β (cid:48) , r ) , then it is a 2 εγ (cid:107) r (cid:107) ∞ / (1 − γ ) -optimal policy of ( S , A , O , α, β, r ) .
Hence, if β does not fulﬁll the invertability assumption made in Theorem 20 an arbitrary small perturbation of it does (given that β is a square matrix) and hence Theorem 20 provides an upper bound on the number of critical points of an approximate problem.
Further, note that the faces, which are guaranteed to contain an optimal policy by (Mont ´ ufar & Rauh, 2017) might be considerably fewer for the
POMDP ( S , A , O , α, β (cid:48) , r ) .
The bound equation 6 could be used to identify the best perturbations of a given magnitude to obtain a problem with a minimal number of critical points.
Remark 65 (Design of policy models) .
Knowledge about the location of critical points of the reward function can be used to design policy models, which provably include those critical points and therefore also the optimal policy.
D. N UMBER OF CRITICAL POINTS IN A TWO - ACTION BLIND CONTROLLER
This subsection is devoted to the proof of Proposition 21 that we restate here for convenience.
Proposition 21 (Number of critical points in a blind controller) .
Let ( S , O , A , α, β, r ) be a POMDP describing a blind controller with two actions, i.e., O = { o } and A = { a 1 , a 2 } and let r, α and µ be generic and let γ ∈ (0 , 1) .
Then the reward function R µγ has at most |S| critical points in the interior int(∆ OA ) ∼ = (0 , 1) of the policy polytope and hence at most |S| + 2 critical points.
Before we present the proof of this result, we discuss how the bound on the rational degree of the reward function leads to am upper bound on the number of critical points.
We consider a blind controller and restrict ourselves to the discounted case γ ∈ (0 , 1) .
We associate the policy polytope ∆ OA with [0 , 1] and for p ∈ [0 , 1] we write π p and η p for the associated policy and the state-action frequency.
From Theorem 4 we know that the reward function R = f/g : [0 , 1] → R is a rational function of degree at most k := |S| , which is well known to possess at most 2 k − 2 critical points.
Hence, there are at most this many critical points in the interior (0 , 1) if the reward function is not constant.
Now we use the geometric description of the set of state-action frequencies and yields a reﬁned bound.
Proof of Proposition 21.
First, we note that since µ is generic and γ < 1 Assumption 13 is satisﬁed.
In this case, the combinatorial part is simple, since there are only two zero-dimensional faces of the state-action frequencies (corresponding to the endpoints of the unit interval) and one one-dimensional face (corresponding to the interior of the unit interval).
Let us set
U = {
[Equation]
∈ R S×A | η T w sγ = (
[Equation]
− γ
[Equation]
µ s for all s ∈ S} , where w sγ := δ s ⊗ 1 − γα ( s |· , · ) .
By Proposition 8 and Example 15 the set of discounted state-action frequencies is given by
[Equation]
Like above, we associate the policy polytope ∆ OA with [0 , 1] and for p ∈ [0 , 1] we write π p and η p for the associated policy and the state-action frequency.
We aim to bound the number of critical points of the reward function over (0 , 1) or equivalently the number of critical over { η p | p ∈ (0 , 1) } where we used that Assumption 13 holds.
Further, recall that η p ( a | s ) =
η psa /ρ ps , we have that
[Equation]
Thus the number of critical points over { η p | p ∈ (0 , 1) } are upper bounded by the number of critical points on U ∩D 1 .
Note that if α and µ are generic, the subspace U is in general position.
Further, its dimension is |S||A| − |S| = |S| , where we used |A| = 2 .
Hence, the number of complex solutions to the KKT conditions over U ∩ D 1 are given by the k -th polar degree δ k
( D 1 ) , where k := |S| , where we also used the genericity of the reward vector.
We can compute the polar degree using the formula presented by ¨Ozl¨um C¸elik et al.
(2021) to obtain
[Equation]
(cid:88) s
= ( − 1) s (cid:18) k − s +k −(cid:19) ( k − s )!
  (cid:88) i + j = s (cid:0) ki (cid:1) ( k − 1 − i )!
· (cid:0) 2 j (cid:1) (2 − 1 − j )! 
 .
We calculate the three individual terms to be (cid:18) k
+k −(cid:19) k !   (cid:88)
i + j = (cid:0) ki (cid:1) ( k − 1 − i )!
· (cid:0) 2 j (cid:1) (2 − 1 − j )! 
 =
( k + 1) k · k ! · (cid:0) k 0 (cid:1) ( k − 1)!
· (cid:0) 20 (cid:1) 1!
=
( k + 1) k , and − (cid:18) k k −(cid:19)
( k − 1)!
(cid:32) (cid:0) k 1 (cid:1) ( k − 2)!
+ (cid:0) 21 (cid:1) ( k − 1)!
(cid:33) =
− k !
(cid:18)
k ( k − 2)!
+ ( k − 1)!
(cid:19)
= − k 2 ( k − 1) − 2 k, and (cid:18) k −k −(cid:19)
( k − 2)!
(cid:32) (cid:0) k 1 (cid:1)(cid:0) 21 (cid:1) ( k − 2)!
+ (cid:0) k 2 (cid:1) ( k − 3)!
(cid:33) =
( k − 2)!
(cid:18) 2 k ( k − 2)!
+
[Equation]
Adding those three summands we obtain
[Equation]
Note that there is also a more structural argument to obtain this polar degree.
In fact, the polar degree δ l ( D 1 ) = 0 for l > dim( D ∗ 1 ) − 1 , where D ∗ 1 denotes the dual variety of D 1 ( ¨Ozl¨um C¸elik et al., 2021).
Note that in the case of k × 2 matrices D ∗
= D 1 (Draisma et al., 2016) and hence it holds that δ l ( D 1 ) =
for l > dim( D 1 ) − 1 = k (Spaenlehauer, 2012).
The largest non-zero polar degree is equal to the degree of the dual variety (Draisma et al., 2016) and hence we obtain δ k
( D 1 )
= degree( D ∗ 1 )
=
degree( D 1 ) =
k (Spaenlehauer, 2012).
Note that this bound is not necessarily sharp, since it is exactly the number of complex solutions of the criticality equations over U ∩D 1 .
Overall, we have seen that the study of the algebraic properties of the reward function provided an upper bound on the number of critical points of the problem, which can be improved using the description of the state-action frequencies as a basic semialgebraic set and employing tools from algebraic geometry.
D. E XAMPLES WITH MULTIPLE SMOOTH AND NON - SMOOTH CRITICAL POINTS
It is the goal of this example to demonstrate that for a blind controller multiple critical points can occur in the interior (0 , 1) ∼ = int(∆ OA ) as well as at the two endpoints of [0 , 1] ∼ = ∆ OA of the policy polytope.
We refer to such points as smooth and non-smooth critical points.
We consider a blind controller with one observation, two actions a 1 , a 2 and three states s 1 , s 2 , s 3 and a deterministic transition kernel α and reward described by the graph shown in Figure 2.
[Figure]
We make the usual identiﬁcation [0 , 1] ∼ = ∆ OA , where we associate p with π ( a 1 | o ) .
In Figure 3, the reward function is plotted on the left for the three initial conditions µ = δ s 1 , δ s 2 , δ s 3 .
It is apparent that the reward has two critical points in the interior of the policy polytope ∆ O A ∼ =
[0 , 1] for the two initial conditions µ = δ s
= δ s 3 .
For µ = δ s 2 , there are two strict local maxima on the two endpoints of the interval.
In this example, the bound from Proposition 21 ensures that there are at most |S| = 3 critical points in the interior and at most |S| +
=
5 critical points in the whole policy polytope.
We see that those bounds are not sharp in this speciﬁc setting.
Note that this example is stable under small perturbations of the transition kernel and reward vector and hence can occur for generic α and r .
The right hand side of Figure 3 shows a three dimensional random projection of the set of feasible discounted state-action frequencies.
By Theorem 4 they are a curve in R S×A ∼ =
Rwith an injective rational parametrization of degree at most |S| = 3 .
[Figure]
D. (S UPER ) LEVEL SETS OF (PO)MDP S D.5. C ONNECTEDNESS OF SUPERLEVEL SETS IN MDP S
Theorem 66 (Existence of improvement paths in MDPs) .
For every policy π ∈ ∆ SA , there is a continuous path connecting π to an optimal policy along which the reward is monotone.
If further π (cid:55)→ η π is injective, the reward is strictly monotone along this path, if π is suboptimal.
In particular, the superlevel sets of MDPs are connected.
Proof.
Let us ﬁx π ∈ ∆ SA and set η 0 :=
η π and η 1 be a global optimum and η t be the linear interpolation and ρ t be the corresponding state marginal.
Note that for s ∈ S it holds that either ρ t ( s ) > 0 for all t ∈ (0 , 1) or ρ t ( s )
=
for all t ∈ [0 , 1] .
In the latter case, we can set π t ( ·| s ) to be an arbitrary element in ∆ A .
For the other states and t ∈ (0 , 1) we can deﬁne the policy through conditioning by π t ( a | s ) :
= η t ( s, a ) /ρ
t ( s ) and will continuously extend the deﬁnition to t ∈ { 0 , 1 } in the following.
If ρ 0 ( s ) > 0 or ρ 1 ( s ) > 0 , then the deﬁnition extends naturally.
Suppose that ρ 0 ( s ) = 0 , then we now that ρ 1 ( s ) > 0 since otherwise ρ t ( s ) =
for all t ∈ [0 , 1] .
Now for t >it holds that
[Equation]
which extends continuously to t = 0 .
If ρ 1 ( s ) = 0 , then like before, π t ( ·| s ) does not depend on t and we can extend it to t
= 1 .
Now we have constructed a continuous path π t , such that η π t =
η t and thus
[Equation]
which is strictly increasing if π 0 is suboptimal.
It remains to construct a continuous path between π 0 and π .
Note that if ρ 0 ( s ) > 0 , the policies π 0 and π agree on the state s and so does the linear interpolation between the two policies.
Now, by Proposition 46 we see that every linear interpolation between π 0 and π has the state-action distribution η 0 .
Gluing the two paths, we obtain a path that ﬁrst leaves the state-action distribution unchanged and then increases the reward strictly up to optimality.
D.5. T HE SEMIALGEBRAIC STRUCTURE OF LEVEL AND SUPERLEVEL SETS FOR POMDP S
Consider a POMDP ( S , A , O , α, β, r ) and ﬁx a discount rate γ ∈ (0 , 1) as well as an initial condition µ ∈ ∆ S .
The levelset
[Equation]
of the reward function is the intersection of a variety generated by one determinantal polynomial of degree at most |S| with the policy polytope ∆ OA .
Indeed, by Theorem 4 the reward function R µγ is the fraction f/g of two determinantal polynomials f
and g of degree at most |S| .
The level set consists of all policies, such that f ( π ) =
ag ( π ) .
Thus, the levelset is given by
[Equation]
Analogously, a superlevel set is the intersection
[Equation]
of a basic semialgebraic generated by one determinantal polynomial of degree at most |S| with the policy polytope ∆ OA .
In particular, both the levelset and superlevel sets of POMDPs are semialgebraic sets deﬁned by linear inequalities and equations (corresponding to the conditional probability polytope ∆ OA ) and a determinantal (in)equality of degree at most |S| .
This description can be used to bounds the number of connected components, which captures important properties of the loss landscape of an optimization problem (Barannikov et al., 2019; Catanzaro et al., 2020).
By a theorem due to Łojasiewicz, level and superlevel sets possess ﬁnitely many connected (semialgebraic) components (Ruiz, 1991; Basu et al., 2006) and there exist algorithmic approaches to computing the number of connected components (Grigor’ev & Vorobjov, 1992) as well as explicits upper bounds, which involve the dimension, the number of deﬁning polynomials as well as their degrees (Basu, 2003; 2014).
Those results are generalizations of the classic result due to Milnor and Thom which bounds the sum of all Betti numbers of a variety.
If we apply the Milnor-Thom theorem to the variety V
we obtain that there are at most |S| (2 |S|− 1) |O||A|− 1 many connected components of V .
This bound neglects the determinantal nature of the deﬁning polynomial and might therefore be coarse.
Using an analogue approach, we can also study the level and superlevel sets of the reward function in the space of feasible state-action frequencies.
Indeed, they are the intersections of the hyperplane { η ∈ R S×A | r T η = a } and halfspace { η ∈ R S×A | r T η ≥ a } with the semialgebraic set N µ,βγ of state-action frequencies.
E P OSSIBLE E XTENSIONS E. A PPLICATION TO FINITE MEMORY POLICIES
In general, it is possible to reduce POMDPs with ﬁnite memory policies to a POMDP with memoryless policies by augmenting the state and observation space with the memory.
Say we consider policies with a memory that stores the last k observations that were made.
Then we could set ˜
S :=
S ×
O k − 1 and ˜ O :
= O k .
If the ﬁrst state is s 0 and the ﬁrst observation that is being made is o 0 , then we will associate it with ˜ s 0 :
= ( s 0 , o 0 , . . .
, o 0 ) ∈ ˜ S and ˜ o
:=
( o 0 , . . .
, o 0 ) ∈
˜ O respectively.
If after t steps, the current state is ˜ s t = ( s t , o t − k , . . .
o t − 1 ) and the next observation is o t , then we set ˜ o t :=
( o t − k , . . .
, o t ) .
An analogue strategy can be taken when the memory does consist of more than the history of observations and for example includes the history of decision.
It remains open to explore the implications of the translation of our results to policies with internal memory with this identiﬁcation.
E. P OLYNOMIAL POMDP S
Zahavy et al.
(2021) consider MDPs, where the objective is a convex function of the state-action frequency, i.e., where R µγ ( π ) =
f ( η π,µγ ) for some convex function f : R S×A → R and coin the name of convex MDPs .
In analogy, we refer to the case where f is a polyomial function as polynomial (PO)MDPs .
In polynomial POMDPs, the problem of reward maximization is by deﬁnition an optimization problem of a polynomial function over the set N µ,βγ of feasible state-action frequencies.
Since the feasible state-action frequencies form a basic semialgebraic set, the problem of reward maximization in polynomials is a polynomial optimization problem.
Hence, the method of bounding the number of critical points as discussed in Section 5 generalizes to the case of polynomial reward criteria.
If f is a polynomial of degree d , the upper bound (6) from Theorem 20 takes the form
[Equation]
The use of polar degrees does not extend in general to the case of polynomial POMDPs, since they require a linear objective function, but can still be related to the algebraic degree for a quadratic objective as it is the case for the Euclidean distance function (Draisma et al., 2016).
F E XAMPLES
Here, we provide examples, which illustrate our ﬁndings.
In particular, we compute the deﬁning polynomial inequalities of the set of feasible state-action frequencies for the example from Figureand a navigation problem in a grid world.
We use polynomial optimization algorithms to solve thie polynomial programming formulation of POMPDs and see that in this offers a computationally feasible approach to the reward maximization problem.
F. T OY EXAMPLE OF F IGURE
We discuss in detail a toy POMDP which we used to generate the plots in Figure 1.
We consider state, observation, and action spaces with two elements each, as well as following deterministic transition mechanism α , observation mechanism β , and instantaneous reward r :
[Equation]
[Equation] [Equation]
The transitions, instantaneous rewards, and observations are shown in Figure 4.
As an initial distribution we take the uniform distribution µ =
( δ s
+ δ s 2 ) / 2 over the states.
[Equation]
a
[Equation] [Equation]
ssoo 1 /1 /
Polynomial programming formulation To illustrate Theorem 16 (and Proposition 14), we derive step-by-step the explicit polynomial program for the reward maximization in this toy example.
For this, we ﬁrst compute the deﬁning inequalities of the set of feasible state-action frequencies.
We begin with the linear constraints that deﬁne the set N µγ of state-action frequencies of the associated
MDP, given in general form in Proposition 8.
Further, we denote the state-action frequencies as matrices
[Equation]
Following Proposition 8, the linear inequalities are η ij ≥
for all i, j ∈ { 1 , 2 } , and the linear equations are η T w iγ =
(1 − γ ) µ i = 1 / 4 for i ∈ { 1 , 2 } , whereby here
[Equation]
and
[Equation]
Thus the two linear equations are
[Equation]
It remains to compute the polynomial inequalities, which can be done using Remark 18.
We invert the matrix β and obtain
[Equation]
Using the notation from Remark 18 we have S o
= { s 1 } and S o
= { s 1 , s 2 } , and thus the polynomial inequalities are
[Equation]
The ﬁrst two inequalities can be seen to be redundant and can be discarded.
Finally, note that the objective function is given by
[Equation]
Hence
[Equation]
we have obtained the following explicit formulation of the reward maximization problem as a polynomial optimization problem:
[Equation]
subject to
[Equation]
Solution with constrained and polynomial optimization tools The formulation (30) allows us to use polynomial optimization algorithms, semi-deﬁnite programming (SDP) solvers, or relaxation hierarchies such as the popular Sum Of Squares (SOS).
Using the JuMP and Ipopt Julia packages we directly obtained the globally optimal solution to problem (rounded to three digits)η
[Equation]
167 0 .(cid:19) .
The corresponding optimal state policy τ ∗ is obtained simply by conditioning on states, and any pre-image under the observation kernel is an optimal observation policy, in this case simply π ∗ =
β − 1 τ ∗ ,
[Equation]
This policy achieves a reward of R µγ ( π ∗ ) =
(cid:104) r, η ∗ (cid:105) S×A = 0 .
833 (rounded to three digits).
The computations took 0 .
01 s (on a 2 GHz Quad-Core Intel Core i5 processor).
The command in JuMP to call the optimizer Ipopt is simply: model =
Model(optimizer_with_attributes(Ipopt.
Optimizer) @variable(model, \eta[1:2, 1:2]>=0) @constraint(model, 2\eta[1, 1] + 4\eta[1, 2] - 2\eta[2, 1]
== 1) @constraint(model, -2\eta[1, 2] + 4\eta[2, 1] + 2\eta[2, 2] =
= 1) @constraint(model, \eta[1, 1]\eta[2, 1] + 2\eta[2, 1][1, 2] - \eta[1, 1]\eta[2, 2] >= 0) @constraint(model, \eta[1, 2]\eta[2, 2] + 2\eta[1, 1]\eta[2, 2] - \eta[1, 2]\eta[2, 1] >= 0) @NLobjective(model, Max, \eta[1, 1] + \eta[2, 2])
optimize!(model)
For completeness, we also provide the command to solve a relaxation in Python SumOfSquares , which is the following, although we found this to run a bit slower depending on the selected degree.
Here we negate the objective in order to obtain a minimization problem and square the search variables (which are required to be non-negative) in order to obtain polynomials of even degree: e11, e12, e21, e22 = sp.symbols(’e11 e12 e21 e22’) prob =
poly_opt_prob([e11, e12, e21, e22], - e11**2 - e22**2, eqs=[+ 2 * e11**2 + 4 * e12**2 - 2 * e21**2 - 1, - 2 * e12**2 + 4 * e21**2 + 2 * e22**2 - 1, + e11**2 + e12**2 + e21**2 + e22**2 - 1], ineqs=[e11**2 * e21**2 + 2 * e21**2 * e12**2 - e11**2 * e22**2, e12**2 * e22**2 + 2 * e11**2 * e22**2 - e12**2 * e21**2], deg=2)
prob.solve() print(prob.value)
Policy gradient methods may not ﬁnd a global optimum We want to demonstrate an important problem of policy gradient methods, which is the well known possibility to get stuck in local optima, in the case of the toy example.
For this, we used a tabular softmax policy model to represent the interior of the policy polytope ∆ OA , i.e. used the following parametric policy model
[Equation]
for θ ∈ R O×A .
We computed 15 policy gradient trajectories, where we used the policy gradient theorem (see Corollary 52) to compute the update directions.
The starting positions where generated randomly, such that the initial conditions in the policy polytope ∆ OA are uniformly random.
The trajectories in the policy polytope ∆ O A ∼ =
[0 , 1] 2 are shown in Figure 5, which also shows a heat map of the reward function.
We observe that 5 of the trajectories converge to a suboptimal strict local minimum.
Note that this is not artefact of the parametrization, but of the fact that there is a strict local minimum and hence every local optimization method will suffer from this problem.
The reward of the suboptimal local minimum
[Equation]
[Figure]
R is 0 .
747 if rounded to 3 digits.
Number of critical points We evaluate the bound of Theorem 20 for this toy problem.
First note that in this example the observation matrix β is invertible with β − 1 = (cid:0) 1 0 − 1 2 (cid:1) .
Further, Assumption 13 is satisﬁed for initial distributions µ with full support.
Hence, we can apply Theorem 20.
Here, we have |S| = |A| = |O|
= 2 and in the notation of Theorem 20 we have d o 1 = 1 and d o 2 = 2 .
As discussed in the main body, the bound evaluates to zero if we consider the interior of the policy polytope, which corresponds to I = ∅ .
This means that there are no critical points in the interior of the policy polytope, in other words, all optimal policies lie at the boundary and hence have one or more zero entries.
The one-dimensional faces correspond to the index sets { ( a 1 , o 1 ) } , { ( a 2 , o 1 ) } , { ( a 1 , o 2 ) } , { ( a 2 , o 2 ) } .
The choices I = { ( a 1 , o 1 ) } , { ( a 2 , o 1 ) } correspond to the two edges on the left and right of the policy polytope ∆ OA as shown in the top left corner of
Figure 1 or alternatively to the two straight faces of the set N µ,βγ of state-action distributions shown in the top right corner.
The bound (6) evaluates to zero for those choices.
This can also be seen in the bottom row in Figure 1, where it is apparent that there are no critical points on the respective faces.
For the choices I = { ( a 1 , o 2 ) } , { ( a 2 , o 2 ) } the bound (6) evaluates to two.
Indeed, these faces contain critical points.
The bound is not sharp in this case since the actual number of critical points in any of the two faces of the policy polytope ∆ OA , which correspond to the two non-linear faces of
N µ,βγ is one.
Nonetheless, this illustrates how the theorem allows us to discard most faces of the polytope and focus the search for an optimal policy on just two faces.
F. N AVIGATION IN A GRID WORLD
We consider a grid world depicted in Figure 6 with 13 states and 7 observations.
The four actions are { R, L, U, D } corresponding to the directions right, left, up and down on the grid and we denote them with { 1 , 2 , 3 , 4 } .
The transitions are deterministic and lead to the cell right, left, above or below the current cell, if this cell is admissible.
Further, we consider deterministic observations, which correspond to the agent being able to observe its immediate four neighboring positions.
This observation mechanism partitions the state space into the seven subsets { 1 , 7 } , { 2 , 4 , 9 , 10 } , { 3 , 8 } , { 5 } , { 6 , 12 } , { 11 } , { 13 } , which lead to the observations o 1 , o 2 , o 3 , o 4 , o 5 , o 6 and o 7 respectively.
Hence, by Remark 59 the polynomial constraints are given by
[Equation]
[Figure]
[Equation]
where ρ s = (cid:80) a η sa .
The linear constraints apart from η ≥ 0 can be computed to be
[Equation]
[Equation] [Equation]
η
[Equation]
η
[Equation]
η
[Equation]
µ
[Equation]
8 − γ
[Equation]
η 71 + η 83 + η 84 + η 92 )
=
µρ 9 − γ
[Equation]
η 81 + η 93 + η 10 , 2 ) =
µρ 10 − γ ( η 91 + η 10 , 3 + η 10 , 4 + η 11 , 2 ) =
µρ 11 − γ ( η 10 , 1 + η 11 , 1 + η 11 , 4 ) =
µρ 12 − γ ( η
[Equation]
Further, the objective function is given by
[Equation]
Let us now consider the uniform distribution µ s = 1 / 13 for s ∈ S as an initial distribution and γ = 0 .
999 as a discount factor.
Like for the toy problem we used the interior point method Ipopt implemented in the Julia packages JuMP and Ipopt to solve this polynomial optimization problem.
The solver took around 0 .
03 s consistently (on a 2 GHz Quad-Core Intel Core i5 processor).
The
[Table]
and has the objective value (cid:104) r, η ∗ (cid:105) S×A = 0 . 033 .
The corresponding optimal state policy τ ∗ is obtained simply by conditioning on states, and any pre-image under the observation kernel is an optimal observation policy, in this case simply π ∗ =
β + τ ∗ which is (rounded to two digits)
[Equation]
[Table]
[Equation]
[Table]
This policy
moves right on observation 1 corresponding to the states 1 and 7 , 2.
moves right and left with probability close to 1 / 2 on observation 2 corresponding to states 2 , 4 , 9 and 10 , 3.
moves right and left with probability close to 1 / 2 on observation 3 corresponding to the states 3 and 8 , 4.
moves left on observation 4 corresponding to the state 5 , 5. moves up on observation 5 corresponding to the states 6 and 12 , 6.
moves up on observation 6 corresponding to the states 11 , 7.
moves up on observation 7 corresponding to the state 13 .
The action choices of the policy π ∗ are also shown in Figure 7.
Note that the policy ˆ π selects the best action in the states 1 , 5 , 6 , 11 , 12 and 13 .
Those are the states that are either identiﬁable from its observation (this is the case for 5 , 11 and 13 ) or where the optimal actions of all states leading to the same observation agree (this is the case for the pairs { 1 , 7 } and { 6 , 12 } ).
In the other states, where the corresponding observation is ambiguous, the policy randomizes among the two actions, which are optimal for the compatible states.
This is for example the case for the states 2 , 4 , 9 and 10 , which all lead to observation 2 .
The optimal MDP policy would move left in state 2 and 4 and move right in the states 9 and 10 .
The POMDP policy has to randomize between moving left and right, since otherwise the agent could never reach the goal state if starting in 7 or 13 .
The same consideration applies to the states 3 and 8 , which both lead to observation 3 .
The Julia code is available in the supplements.
F. A THREE DIMENSIONAL EXAMPLE
Let us now discuss an example where the set of discounted state-action distributions is threedimensional and not two-dimensional as before.
For this, we consider a generalization of the previous example where |S| = 3 , |A| = 2 and |O| =
3 such that
[Equation]
→ ↔ ↔ ↔ ← ↑ → ↔ ↔ ↔ ↑
[Figure]
The observation mechanism used is
[Equation]
Further, the action mechanism α and the initial distribution µ are sampled randomly and the used discount factor is 0 .
Since the initial distribution is generic and β is invertible, the set of stateaction frequencies N µγ and the set of feasible state-action frequencies N µ,βγ are three-dimensional and are in fact combinatorially equivalent to the three-dimensional cube ∆ SA ∼ = ∆ OA ∼ =
[0 , 1] 3 (see
Theorem 16).
In Figure 8 we plot a random three-dimensional projection of the sets.
More precisely, we plot their one-dimnesional faces dashed and solid for the MDP and POMDP respectively.
The combinatorial equivalence to the three-dimensional cube can be see in this plot.