Learning Generalized Policy Automata for Relational Stochastic Shortest Path Problems
Abstract
Several goal-oriented problems in the real-world can be naturally expressed as
Stochastic Shortest Path problems (SSPs).
However, the computational complexity of solving SSPs makes  nding solutions to even moderately sized problems intractable.
State-of-the-art SSP solvers are unable to learn generalized solutions or policies that would solve multiple problem instances with different object names and/or quantities.
This paper presents an approach for learning Generalized Policy
Automata (GPAs): non-deterministic partial policies that can be used to catalyze the solution process.
GPAs are learned using relational, feature-based abstractions, which makes them applicable on broad classes of related problems with different object names and quantities.
Theoretical analysis of this approach shows that it guarantees completeness and hierarchical optimality.
Empirical analysis shows that this approach effectively learns broadly applicable policy knowledge in a few-shot fashion and signi  cantly outperforms state-of-the-art SSP solvers on test problems whose object counts are far greater than those used during training.
Introduction
Goal-oriented Markov Decision Processes (MDPs) expressed as Stochastic Shortest Path problems (SSPs) have been the subject of active research since they provide a convenient framework for modeling the uncertainty in action execution that often arises in the real-world.
Recently, research in deep learning has demonstrated success in solving goal-oriented MDPs using image-based state representations (Tamar et al., 2016; Groshev et al., 2018).
However, such methods require signi  cant human-engineering effort in  nding transformations like grayscale conversion, etc., to yield representations that facilitate learning.
Many practical problems however, are more intuitively expressed using relational representations and have been widely studied in the literature
As an example, consider a planetary rover whose mission is to collect all rocks of interest from a planet’s surface and deliver them to the base for analysis.
Such a problem objective is not easily described in an image-based representation (e.g., visibility is affected by line of sight) but can be easily described using a relational description language such as  rst-order logic.
Finding suitable imagebased representations for such problems would be counter-productive and dif  cult.
Furthermore, image-based deep learning methods often require large amounts of training data and/or are unable to provide guarantees of completeness and/or convergence.
Many real-world problems such as the rover example above can be readily expressed as SSPs using symbolic descriptions that can be solved in polynomial time in terms of the state space.
SSP algorithms use a combination of pruning strategies (e.g., heuristics (Hansen and Zilberstein, 2001)) that can eliminate large parts of the search space from consideration thereby reducing the computational effort expended.
In spite of such optimizations, a major hurdle is the “curse-of-dimensionality” since the state spaces grow exponentially as the total number of objects increases.
Existing SSP solvers would have dif  culty scaling to rover problems with many locations and/or rocks.
The pruning strategies employed by these SSP solvers do not scale well because they do not use knowledge that could have been exploited from solving similar problems.
One solution to this problem is to compute a simple generalized policy : move the rover to the closest available location with an interesting rock, try loading the rock until it succeeds, navigate back to the base, unload it, and reiterate this process until all the interesting rocks are at the base.
This generalized policy can be used to solve any rover instance with larger numbers of locations and rocks sharing a similar goal objective.
Related work in Generalized Planning addresses the problem of computing generalized policies by learning reliable controllers for broad classes of problems (Srivastava et al., 2008; Bonet et al., 2009;
Aguas et al., 2016).
Recently, Deep Learning based approaches have been shown to successfully learn generalized policies (Toyer et al., 2018;
Garg et al., 2020; Karia and Srivastava, 2022).
A key limitation of these approaches is the lack of any theoretical guarantees of  nding a solution or optimality.
In this paper, we show that such policies can be learned with guarantees of completeness and hierarchical optimality using solutions of very few, small problems with few objects.
The primary contribution of this paper is a novel approach for few-shot learning of Generalized
Policy Automata (GPAs) using solutions of SSP instances with small object counts.
GPAs are nondeterministic partial policies that represent generalized knowledge that can be applied to problems with different object names and larger object counts.
This process uses logical feature-based abstractions to lift instance-speci  c information like object names and counts while preserving the relationships between objects in a way that can be used to express generalized knowledge.
GPAs learned using our approach can be used to accelerate any model-based SSP solver by pruning out large sets of actions in different, related, but larger SSPs with many objects.
We prove that our approach is complete and guarantees hierarchical optimality.
Empirical analysis on a range of well-known benchmark domains shows that our approach few-shot learns GPAs using as few as three training problem instances and convincingly outperforms existing state-of-the-art SSP solvers and does so without compromising the quality of the solutions found.
The rest of this paper is organized as follows: The next section provides the necessary background.
Sec. 3 describes our approach for using example policies in conjunction with abstractions to learn
GPAs and using them for solving SSPs.
We present our experimental setup and discuss obtained results in Sec. 4.
Sec. 5 provides a description of related work in the area.
Finally, Sec. 6 states the conclusions that we draw upon from this work followed by a brief description of future work.
Background
Our problem setting considers SSPs expressed in a symbolic description language such as the
Probabilistic Planning Domain De  nition Language (PPDDL) (Younes et al., 2005).
Let D = ⟨ P , A ⟩ be a problem domain where P and A are  nite sets of predicates and parameterized actions.
Object types , such as those used in PPDDL, can be equivalently represented using unary predicates.
A relational SSP problem instance for a domain D with a goal formula g over P and a  nite set of objects O is de  ned as a tuple P = ⟨ O, S, A, T, C, s 0 , g ⟩ .
A fact is the instantiation of a predicate p ∈ P with the appropriate number of objects from O .
A state s is a set of true facts and the state space S is de  ned as all possible sets of true facts derived using D and O .
Similarly, the action space A is instantiated using A and O .
T : S ×
A × S → [0 , 1] is the transition function and
C : S ×
A × S → R + is the cost function.
An entry t ( s, a, s ′ )
∈
T de  nes the probability of executing action a ∈ A in a state s ∈ S and ending up in a state s ′ ∈ S , and c ( s, a, s ′ )
∈ C indicates the cost incurred while doing so.
Naturally,  s ′ t ( s, a, s ′ )
= 1 for any s ∈ S and a ∈ A .
Note that a refers to the instantiated action a ( o 1 , . . .
, o n ) ,
where o 1 , . .
, o n ∈ O are the action parameters .
We omit the parameters when it is clear from context.
s 0 ∈ S is a known initial state.
A goal state s g is a state such that s g | = g .
For any action a ∈ A and any goal state s g ∈ S , c
( s g , a, s g ) = 0 and t ( s g , a, s g ) = 1 .
Additionally, termination (reaching a state s such that s | = g ) in an SSP is inevitable making the length of the horizon unknown but  nite (Bertsekas and Tsitsiklis, 1996).
Running example: The planetary rover example can be expressed using a domain that consists of predicates connected ( l x , l y ) , interesting ( r x ) , rover-at ( l x ) , in-rover ( r x ) , rock-at ( r x , l x ) , and parameterized actions load ( r x , l x ) , unload ( r x , l x ) , and move ( l x , l y ) .
Object types can be denoted using unary predicates location ( l x ) and rock ( r x ) .
l x , l y , and r x are parameters that can be instantiated with different locations and rocks thus allowing for an easy way to express different problems.
Action dynamics are described using closed-form probability distributions (e.g., loading a rock could be modeled so that the rover picks up the rock with a probability of 0.8) and this forms the transition function.
A simpli  ed SSP problem that ignores connectivity and consists of two locations, a base location, and two rocks can be described using a set of objects O = { l 1 , l 2 , l base , r 1 , r 2 } .
A state in this SSP s eg that describes the situation where r 2 is being carried by the rover and r 1 is at l 2 can be written as s eg = { location ( l 1 ) , location ( l 2 ) , location ( l base ) , rock ( r 1 ) , rock ( r 2 ) , in-rover ( r 2 ) , rock-at ( r 1 , l 2 ) } .
The goal of delivering all the rocks to the base can be expressed as ∀ x rock ( x ) =
⇒ rock-at ( x, l base ) .
Executing any action can be assumed to expend some fuel and as a result, the objective is to deliver all the rocks to the base in a way that minimizes the total fuel expended.
A solution to an SSP is a deterministic policy π : S → A , that is a mapping from states to actions.
A proper policy is one that is well-de  ned for all states.
A complete proper policy is one for which termination is guaranteed from all possible states.
By de  nition, SSPs must have at least one complete proper policy (Bertsekas and Tsitsiklis, 1996).
This can be overly limiting in practice since such a formulation does not model dead end states: states from which the goal is reachable with probability 0.
A weaker formulation of an SSP stipulates that the goal must be reachable with a probability offrom s 0 i.e. whose solution is a partial proper policy that is well-de  ned for every state reachable from s 0 .
As such, we focus on a broader class of relaxed SSPs called Generalized SSPs (GSSPs) (Kolobov et al., 2012) that only require the existence of at least one partial proper policy.
Henceforth, we use the term SSPs to refer to GSSPs and focus only on partial proper policies.
The value of a state s when using a policy π is the expected cost of executing π ( s ) when starting in s and following π thereafter: V π ( s )
=
 s
′ ∈ S t ( s, π ( s ) , s ′ )
[ c ( s, π ( s ) , s ′ )
+ V π ( s ′ )]
(Sutton and
Barto, 1998).
Naturally, V π ( s 0 ) = ∞ if π is not a partial proper policy.
V is known as the value function.
V ∗ is the optimal value function and can be described using the Bellman optimality equation :
[Equation]
In this paper, we focus on optimality w.r.t.
a given initial state.
The optimal policy π ∗ for an SSP P w.r.t.
s 0 is a policy that is better than or equal to all other policies i.e. V π ∗ ( s 0 )
≤ V π ( s 0 ) for any policy π .
SSP solvers iteratively apply the Bellman optimality equation starting from s 0 to compute a policy, and under certain conditions, have been proved to converge to a policy that is ϵ -consistent with π ∗ (Hansen and Zilberstein, 2001; Bonet and Geffner, 2003).
Let F α and F β be two sets of features.
We use feature-based abstractions to lift problem-speci  c characteristics like object names and numbers in order to facilitate the learning of generalized knowledge that can be applied to problems irrespective of differences in such characteristics.
Given any SSP P , we de  ne state abstraction as a function α : F α , S → S that transforms the concrete state space S of P into a  nite abstract state space S .
Similarly, action abstraction β : F β , S, A
→ A transforms the action space to a  nite abstract action space.
Typically, | S | ≤ | S | and | A | ≤ | A | .
We use s =
α ( F α , s ) and a = β ( F β , s, a ) to represent abstractions of a concrete state s and action a .
In this paper, we utilize feature sets automatically derived using canonical abstraction (Sagiv et al., 2002) to compute such feature-based representations of s and a .
This is described in Sec. 3.1.
Our Approach
Our objective is to exploit knowledge from solutions of SSP instances with small object counts to learn Generalized Policy Automata (GPAs) that allow effective pruning of the search space of related
SSPs with larger object counts.
We accomplish this by using solutions to a small set of training instances that are easily solvable using existing SSP solvers, and using feature-based canonical abstractions to learn a GPA that encodes generalized partial policies and serves as a guide to prune the set of policies under consideration.
We provide a brief description of canonical abstraction in
Sec. 3.1, de  ne GPAs in Sec. 3.2, and describe our process to learn a GPA in Sec.
We then describe our method (Alg. 1) for solving SSPs and state its theoretical properties in Sec. 3.3.
3.1 Canonical Abstraction
Canonical abstractions, commonly used in program analysis, have been shown to be useful in generalized planning (Srivastava et al., 2011; Karia and Srivastava, 2021).
Canonical abstractions group together multiple objects in a state using object roles.
Given a concrete state s and an object o , the set of unary predicates that o satis  es is known as the role of o .
0-ary predicates are represented as unary predicates with a default ‘phantom’ object.
Let ψ be a role.
Then we de  ne φ ψ ( s ) as a function that returns the set of objects that map to ψ in a concrete state s .
Similarly, for any given predicate p n ∈ P where n is the arity, φ p n ( ψ 1 ,..., ψ n ) ( s ) is de  ned as the set of all n -ary predicates in s that are consistent with the roles composing the predicate p n ( ψ 1 , . . .
, ψ n ) , i.e., φ p n ( ψ 1 ,..., ψ n ) ( s ) =
{ p n ( o 1 , . . .
, o n ) |
p n ( o 1 , . . .
, o n ) ∈ s, o
i ∈
φ ψ i ( s ) } .
The value of a role ψ in a concrete state s is given as max(2 , | φ ψ ( s ) | ) to indicate whether there are 0, 1, or more than 1 objects satisfying ψ .
Since relations between objects become imprecise when grouped as roles, the value of p n ( ψ 1 , . . .
, ψ n ) in s is determined using three-valued logic and is 0 if φ p n ( ψ 1 ,..., ψ n ) ( s ) = {} , 1 if | φ p n ( ψ 1 ,..., ψ n ) ( s ) | = | φ ψ 1 ( s ) | × . . .
× | φ ψ n ( s ) | , and 12 otherwise.
Let Ψ be the set of all possible roles and P i be the set of predicate names p ∈ P of arity i for a domain D , then P i
= P i ×
[ Ψ ] i is the set of all possible relations of arity i between roles.
We de  ne the feature set for state abstraction as F α = Ψ ∪ Ni =2 P i where N is the maximum arity of any predicate in D .
We de  ne state abstraction α
( F α , s ) for a given concrete state s to return an abstract state s as a total valuation of F α using the process described above.
Similarly, we de  ne the feature set for action abstraction as F β = Ψ .
The action abstraction β ( F β , s, a ) for a concrete action a ( o 1 , . . .
, o n ) when applied to s returns an abstract action a ( ψ 1 , . . .
, ψ n ) where the action name a ≡ a and ψ i is the role of the parameter o i ,
i.e., o i ∈
φ ψ i ( s ) for any ψ i ∈ Ψ .
Fig.
1 provides an intuitive example of how canonical abstraction can be used to lift instance-speci  c information.
The  gure describes two concrete states s 1 and s 2 from two different problems.
scontains 2 rocks and 3 locations whereas s 2 contains 6 rocks and 4 locations.
There are four roles in s .
For example, in s 1 , r 1 maps to the the role for a rock ψ 1 , and r 2 maps to the role for an interesting rock ψ 2 , i.e., φ ψ 1 ( s 1 ) =
{ r 1 } and φ ψ 2 ( s 1 ) = { r 2 } .
The abstract relation rock-at ( ψ i , ψ j ) provides the three-valued representation of values between different roles.
For example, rock-at ( ψ 1 , ψ 3 ) is interpreted as the set of rocks that are at some location , while rock-at ( ψ 2 , ψ 4 ) is interpreted as the set of interesting rocks that are at the base .
Since rock-at ( r 1 , l 2 ) does not appear in s 1 , rock-at ( ψ 1 , ψ 3 ) evaluates to 0 .
5 to indicate that there are some rocks that are at some, but not all, locations .
Similarly, since all interesting rocks are at the base location, rock-at ( ψ 2 , ψ 4 ) evaluates to 1.
The key aspect of abstraction comes from the observation that the relation rock-at ( ψ i , ψ j ) remains the same for s 2 even though s 2 has many more objects than s 1 .
The same high-level interpretations of the relations are captured while lifting low-level information like object names and numbers.
3.2 Generalized Policy Automata
We introduce Generalized Policy Automata (GPAs) as compact and expressive non-deterministic  nitestate automata that encode generalized knowledge and can be represented as directed hypergraphs.
GPAs impose hierarchical constraints on the state space of an SSP and prune the action space under consideration, thus reducing the computational effort of solving larger related SSP instances.
De  nition 3.1 (Generalized Policy Automaton) .
Let S and A be a set of abstract states and actions.
A Generalized Policy Automaton (GPA) G = ⟨ V , E ⟩ is a non-deterministic  nite-state automaton that can be represented as a directed hypergraph where the set of vertices V = S .
E ⊆ V × P ( V ) \ ∅ × A , where P ( V ) is the powerset of V , is a set of directed hyperedges s.t.
each hyperedge e ∈
E is a tuple ( e src , e dest , e act ) representing a start vertex, a set of result vertices, and an action label.
De  nition 3.2 (GPA Consistent Policy) .
A policy π for an SSP P is de  ned to be consistent with a
GPA G iff for any states s, s ′ ∈ S and any action a ∈ A
whenever π ( s ) = a and t ( s, a, s ′ ) >
there exists a hyperedge e ≡ ( e src , e dest , e act ) ∈ E where s = e src , s
′ ∈
e dest , and a = e act .
De  nition 3.3 (Hierarchically Optimal Policy) .
A policy π given a GPA G is hierarchically optimal for an SSP P iff the expected value of π is minimal among all possible GPA consistent policies, i.e.,
V π ( s 0 ) ≤ V π ′ ( s 0 ) for any GPA consistent policy π ′ .
3.2.1 Learning GPAs
[Figure]
It is well-known that solutions to small problems can be used to construct generalized control structures that can assist in solving larger problems.
We adopt a similar strategy of the learn-from-small-examples approach (Wu and Givan, 2007; Karia and Srivastava, 2021, 2022) and compute GPAs iteratively from a small training set containing solutions of similar SSP instances.
To form our training set, we use a library of solution policies Π
= { π 1 , . . .
, π n } for small SSPs
P 1 , . .
, P n that can be easily (and optimally) computed by existing
SSP solvers.
We use the transition function for P i to convert each policy π i ∈ Π to a set of transitions τ i = { ( s, a, s ′ ) | s, s ′ ∈ S
i , a ∈
A i , π i ( s ) =
a, t i ( s, a, s ′ ) > 0 } .
We then construct our training set T = τ 1 ∪ . . .
∪ τ n as a set of concrete transitions.
The GPA G Π learned from a  nite set of concrete transitions T is de  ned as follows.
We  rst initialize a GPA G Π = ⟨ {} , {} ⟩ .
Next, we convert T into an abstract transition set
T = { ( α ( F α , s ) , β ( F β , s, a ) , α ( F α , s ′ ))
| ( s, a, s ′ ) ∈ T } .
We then form the vertex set by using all abstract states in T , i.e., V = V ∪ { s, s ′ } for every ( s, a, s ′ ) ∈ T .
Similarly, we convert each abstract transition into a hyperedge and add it to the edge set, i.e. E = E ∪ ( s, { s ′ } , a ) for every ( s, a, s ′ ) ∈ T .
Once this is done, we compress G Π by replacing any edges in E that have the same start nodes and labels but different destinations with a single new edge that combines the destinations of the edges, i.e., for any two edges e 1 , e 2 ∈ E s.t.
e 1 src =
e 2 src , e 1 dest ̸ =
e 2 dest , and e 1 act =
e 2 act ,
E =
E \{ e 1 , e 2 } ∪ ( e 1 src , e 1 dest ∪ e 2 dest , e 1 act ) .
Fig.
2 provides a high-level overview of our procedure for merging GPAs.
Henceforth, we drop the subscript Π from G Π when it is clear from context.
3.3 Solving SSPs using GPAs
The key intuition behind our method is to use the GPA to prune policies that are not consistent with the GPA from the search process.
We accomplish this by solving a GPA constrained problem that satis  es the constraints encoded by the GPA.
We de  ne a GPA constrained problem as follows:
De  nition 3.4 (GPA constrained problem) .
Let P = ⟨ O, S, A, T, C, s 0 , g ⟩ be an SSP for a domain D and let G = ⟨ V , E ⟩ be a GPA.
A GPA constrained problem P | G = ⟨ O, S, A, T, C ′ , s 0 , g ⟩ is de  ned with a cost function C ′ :
S ×
A × S → R + such that C ′
[ s, a, s ′ ]
=
C [ s, a, s ′ ] when there exists a
Require: SSP P = ⟨ O, S, A, T, C, s 0 , g ⟩ , GPA G = ⟨ V , E ⟩ ,
Feature Sets F α , F β , Abstraction Functions α , β 1: C ′ =
C {copy over the cost function of P } 2: for ( s, a, s ′ ) ∈
S ×
A × S do 3: s, a, s ′ ← α
( F α , s ) , β ( F β , s, a ) , α ( F α , s ′ ) 4: if there is no edge e ∈ E s.t.
e src =
s, s ′ ∈
e dest , and e act = a then 5:
C ′ [ s, a, s ′ ]
= ∞ 6: end if 7: end for 8: P ′ =
⟨ O, S, A, T, C ′ , s 0 , g ⟩ 9: V ∗ P ′ ← initializeValueFunction () {Typically using heuristics or randomly} 10: V ∗ P ′ , π ∗ P ′ ← optimallySolveSSP ( P ′ , V ∗ P ′ )
{Using the Bellman optimality equation} 11: if π ∗ P ′ is a partial proper policy then 12: return π ∗ P ′ {Return constrained policy} 13: else 14:
V P , π P ← optimallySolveSSP ( P, V ∗ P ′ )
{Using the Bellman optimality equation} 15: return π P {Return partial proper policy for P } 16: end if hyperedge e ≡ ( e src , e dest , e act ) ∈ E where s = e src , s ′ ∈
e dest , and a = e act and C ′
[ s, a, s ′ ]
= ∞ if there is no such hyperedge.
The goal of modifying the cost function is to prevent concrete transitions whose abstract translations are absent in the GPA from being used when performing Bellman updates for P | G .
Actions belonging to such transitions cannot appear in π ∗ P | G since their costs would be ∞ .
As a result, P | G is not an
SSP and the existence of a partial proper policy is not guaranteed in P | G .
Note that every optimal policy π ∗ P | G for P | G is hierarchically optimal for an SSP P given a GPA G .
Our overall objective is to compute such hierarchically optimal policies in a way such that they are approximately as cost effective as the optimal policy while requiring a fraction of the computational effort.
Given a GPA G , Alg.
1 works as follows.
Lines 1-8 create a GPA constrained problem P ′ .
Next, line 10 optimally solves P ′ using any off-the-shelf SSP solver with a randomly initialized value function (line 9).
If the computed policy π ∗ P ′ is a partial proper policy then it is returned immediately (lines 11-12) else Alg.
1 uses a new instance of the SSP solver to compute a policy for P using V ∗ P ′ as the bootstrapped initial value function.
Information such as whether a state is a dead end, etc., is not carried over.
Line 14 then optimally solves P using V ∗ P ′
as the initial value estimates and returns a partial proper policy π P for P .
Since V ∗ P ′ is only used as an initial bootstrapping estimate for P , an
SSP solver will only return a policy π P that is better than or equal to π ∗ P ′ following standard results on policy improvement for value iteration (Sutton and Barto, 1998).
The following result shows that
Alg.
1 computes hierarchically optimal policies.
Theorem 1.
When Alg.
1 returns a constrained policy π ∗ P ′ (line 12) for an SSP P given a GPA G then π ∗ P ′ is hierarchically optimal.
Proof (Sketch) .
Intuitively, Alg. 1 creates a new problem P ′
(lines 1-8) using the original cost function of P .
The cost function for a concrete transition ( s, a, s ′ )
∈ S ×
A × S is set to ∞ iff the corresponding abstract transition is not part of a hyperedge in G .
As a result, P ′ ≡ P | G , and any partial policy π P ′ for G is a GPA consistent policy.
Since Alg.
1 optimally solves P ′ (line 10), the returned policy is hierarchically optimal.
Alg.
1 computes π ∗ P | G for P | G in the space of cross-product of the states of the GPA G with the states of P , similar to HAMs (Parr and Russell, 1997).
As seen in our empirical analysis in Sec. 4, we observe that a small set of example policies are suf  cient to capture rich generalized control structures that are encoded by such hierarchically optimal policies.
Alg.
1 computes such policies within a fraction of the original computational effort and in most cases with costs comparable to π ∗ P .
Theorem 2.
Given a GPA G and an SSP P , Alg.
1 always returns a partial proper policy.
Proof.
Alg.
1 only returns if it computes a partial proper policy at line 12 or a partial proper policy for P at line 15, which by de  nition always exists.
The next result indicates that the output of Alg. 1 is hierarchically optimal or better in terms of the expected value at s 0 .
Theorem 3.
Let V π be the value function for a policy π returned by Alg. 1 for an SSP P using GPA
G .
Let V ∗ P | G be the optimal value function for P | G , then V π ( s 0 )
≤ V ∗ P | G ( s 0 ) .
Proof.
If Alg. 1  nds a partial proper policy π ∗ P | G for P | G then it returns it immediately (line 11), in which case V π ( s 0 )
=
V ∗ P | G ( s 0 ) .
If π ∗ P | G is not a partial proper policy then V ∗ P | G ( s 0 )
= ∞ .
Since
Alg. 1 is complete (Thm. 2), π is a partial proper policy where V π ( s 0 ) < ∞ .
Corollary 3.1.
If V ∗ P ( s 0 ) =
V ∗ P | G ( s 0 ) , then Alg.
1 returns the optimal policy for P .
The following result indicates that only a  nite set of training examples are needed to learn a GPA such that the constrained problem will always yield the optimal policy for a given domain D .
Theorem 4.
Suppose D is a domain, g is a formula over the predicates in D ’s vocabulary, G Π ∗ is a
GPA s.t.
for every SSP instance P of D whose goal is g , there exists an optimal policy π ∗ P that is consistent with G Π ∗ .
Then there exists a  nite set of policies Π ∗ from which G Π ∗ can be learned.
Proof.
Since the sizes of the abstract state and action spaces S and A are  nite, the size of G Π ∗ is  nite and is bounded by | S | × | A | × | S | .
For every e ∈ E at most | e dest | different transitions are needed to learn e .
Since there is a  nite number of edges, a  nite amount of training data will suf  ce for learning G Π ∗ .
In the worst case, the cost functions of P and P | G are similar and no savings are obtained.
However, in our empirical evaluation, we observed that typically very few and small training problems suf  ce to learn a compact GPA G that allows ef  cient computation of solutions for problems that are signi  cantly larger than those used during training.
Finding the right set of examples Π ∗ from which
G Π ∗ can be learned is an interesting and non-trivial research problem that we leave to future work.
Experiments
We conducted an empirical evaluation on  ve well-known benchmark domains that were selected from the International Planning Competition (IPC) (Long and Fox, 2003), International Probabilistic
Planning Competition (IPPC) (Younes et al., 2005), and robotic planning (Shah et al., 2020).
As a part of our analysis, we use the time required to compute a solution and measure the quality of the solutions found to determine whether GPAs allow ef  cient solving of SSPs.
We chose PPDDL as our representational language, which was the default language in IPPCs until 2011, after which the Relational Dynamic
In  uence Diagram Language (RDDL) (Sanner, 2010) became the default.
We chose PPDDL over RDDL since modern state-of-the-art solvers for PPDDL are available and since RDDL does not allow specifying the goal condition easily.
For our baselines, we focus on complete solvers for SSPs.
Deep Learning based approaches do not guarantee completeness and thus are not directly comparable with our work.
We used Labeled
RTDP (LRTDP) (Bonet and Geffner, 2003) and Soft-FLARES (Pineda and Zilberstein, 2019) which are state-of-the-art (SOA), complete SSP solvers.
These algorithms internally generate their own heuristics for initializing the value function using the input domain and problem  le.
We used the inadmissible FF heuristic (Hoffmann, 2001) as the internal heuristic function for all algorithms since the baselines performed best using it.
We ran our experiments on a cluster of Intel Xeon E5-2680 v4 CPUs running at 2.4 GHz with 16 GiB of RAM.
Our implementation is a Python adaptation of mdp-lib (Pineda and Zilberstein, 2019).
We utilized problem generators from the IPC and IPPC suites and those in Shah et al.
(2020) for generating the training and test problems for all domains.
We provide a brief description of the problem domains below.
Rover ( r, w, s, o )
A set of r rovers need to take images of o objectives, and pickup and drop s samples that are present at one of w waypoints.
This is an IPC domain that was converted to a stochastic version by modifying the pickup action to fail with a probability of 0 .
4 (leaving the state unchanged).
Gripper ( b ) A robot with two grippers is placed in an environment consisting of two rooms A and
B. The objective of the robot is to transfer all the b balls initially located in room A to room B. We modi  ed the gripper to be slippery so that picking a ball has a 20% chance of failure.
Schedule ( C, p ) is an IPPC domain that consists of a set of p packets, each belonging to one of C different classes that need to be queued.
A router must  rst process the arrival of a packet in order to route it.
The interval at which the router processes arrivals is determined by probability 0.94.
Keva ( P, h )
A robot uses P keva planks to build a tower of height h .
Planks are placed in a speci  c order in one of two locations, preferring one location with probability 0 .
Despite this simple setting, the Keva domain has been shown to be a challenging problem in robotics (Shah et al., 2020).
Delicate Can ( c ) An arrangement of c cans on a table of which one is a delicate can.
The objective is to pick up a speci  c goal can.
Cans can obstruct the trajectory to the goal can and need to be moved.
They can be crushed with probability 0.1 (delicate cans with probability 0.8) and need to be revived.
Training Setup Our method learns GPAs in a few-shot fashion requiring little to no training data.
For forming our training set, we used a minimum of three and a maximum of ten different solution policies (obtained using LAO ∗ ) for each domain.
The time required to learn a GPA was less thanseconds in all cases of our experiments.
Our training speed highlights the advantages of GPAs that can be quickly learned in a few-shot setting.
Moreover, compared to neuro-symbolic methods, GPAs are not subject to catastrophic forgetting and new training data can easily be merged with the existing
GPA using the process described in Sec. 3.2.1.
Test Setup We  xed the time and memory limit for each problem to 7200 seconds and 16 GiB respectively.
To demonstrate generalizability, our test set contains problems with object counts that are much larger than the training policies used.
The largest problems in our test sets contain at least twice the number of objects than those used during training.
For example, in the Keva domain, we use training policies with towers of height up to 6 and evaluate on problems with towers of height up to 14.
The minimum and maximum number of problems that we used in our test set are 11 andproblems respectively.
Additional information such as problem parameters, hyperparameters used for con  guring baselines, etc., is included in Appendix B.
4.1 Results and Analysis
Our evaluation metric compares the time required to  nd a partial proper policy.
We also compare the quality of the computed policies by executing the policies for 100 trials with a horizon limit of
100 and averaging the obtained costs.
We report our overall results averaged across 10 different runs and report results up to one standard deviation.
Results of our experiments are illustrated in Fig.
In four out of  ve domains (Schedule, Rover, Keva, and Delicate Can), our approach takes signi  cantly less time compared to the corresponding baseline.
For example, in Schedule, the baselines timed out for all of the large test problems reported.
GPAs are able to successfully prune away action transitions that are not helping, leading to large savings in the computational effort.
The costs obtained for executing these policies are also quite similar to each baseline (e.g., Keva), showing that GPAs are capable of learning good policies much faster without compromising solution quality.
Our approach was unable to outperform the baselines in the Gripper domain.
An interesting phenomenon that we observed was that training policies returned by LAO ∗ were different for the case of even/odd balls due to tie-breaking and this led to the GPA not pruning actions as effectively.
Nevertheless, we expected the GPA to outperform the baselines.
We performed a deeper investigation and found that the FF heuristic used is already well-suited to prune away actions that the GPA would have otherwise pruned.
This results in additional overhead being added in our SSP solver from the process of abstraction.
However, heuristics that allow such pruning are dif  cult to synthesize, and in many cases, are hand-coded by an expert after employing signi  cant effort.
There was a decrease in the gains observed in the Delicate Can domain between problem IDs 10 and 11.
We investigated this issue and found that the GPA constructed was unable to effectively prune transitions in problem ID 11.
This is because the training policies that were used were not suf  cient to learn a GPA that could generalize well to larger problems in the Delicate Can domain.
As a result, π ∗ P | G was not a partial proper policy for problem ID 11 and thus additional computation was required.
Finally, because of the  xed timeout used, the maximum time of the baselines was bounded, making the impact of GPAs appear smaller in larger problems.
For example, in problem ID 25 of the Rover domain, the Soft-FLARES baseline timed out in all our runs, but when allowed to run to convergence, it took over 15000 seconds in a targeted experiment that we performed for investigating this issue.
Related Work
There has been plenty of dedicated research to improve the ef  ciency for solving SSPs.
LAO ∗ (Hansen and Zilberstein, 2001) computes policies by using heuristics to guide the search process.
LRTDP (Bonet and Geffner, 2003) uses a labeling procedure in RTDP wherein a part of the subtree that is ϵ -consistent is marked as solved leading to faster ending of trials.
SSiPP (Trevizan and
Veloso, 2012) uses short-sightedness by only considering reachable states up to t states away and solving this constrained SSP.
Soft-FLARES (Pineda and Zilberstein, 2019) combines labeling and short-sightedness for computing solutions.
These approaches are complete and can be con  gured to return optimal solutions, however, they do not learn any generalized knowledge and as result cannot readily scale to problems with a larger number of objects.
Determinization-based approaches (Yoon et al., 2007; Pineda and Zilberstein, 2014) build sparse representations of SSP problems by reducing the branching factor in the “environment’s choices” (the set of probabilistic effects of an action), while our approach uses abstraction to create abstract controllers that generalize solutions to SSPs and reduces the branching factor in the agent’s choice (the set of applicable actions).
Our approach always considers all possible outcomes of every action.
This is a key advantage of our approach since GPAs are able to better handle unexpected outcomes when executing an action in the policy that would otherwise require replanning.
Boutilier et al.
(2001) utilize decision-theoretic regression to compute generalized policies for  rstorder MDPs represented using situation calculus.
They utilize symbolic dynamic programming to compute a symbolic value function that applies to problems with varying number of objects.
FOALP (Sanner and Boutilier, 2005) uses linear programming to compute an approximation of the value function for  rst-order MDPs while providing upper bounds on the approximation error irrespective of the domain size.
A key limitation of their approach is requiring the use of a representation of action models over which it is possible to regress using situation calculus.
API (Fern et al., 2006) uses approximate policy iteration with taxonomic decision lists to form policies.
They use Monte Carlo simulations with random walks on a single problem to construct a policy.
API offers no guarantees of completeness or hierarchical optimality.
Parr and Russell (1997) propose the hierarchical abstract machine (HAM) framework wherein component solutions from problem instances can be combined to solve larger problem instances ef  ciently.
Recently, Bai and Russell (2017) extended HAMs to Reinforcement Learning (RL) settings − where transition dynamics are not known − by leveraging internal transitions of the HAMs.
A key limitation of both these approaches is that the HAMs were hand-coded by a domain expert.
Related work in Generalized Planning focuses on the problem of computing generalized plans and policies such as our GPAs (Srivastava et al., 2012).
Bonet et al.
(2009) automatically create  nite-state controllers for solving problems using a set of examples by modeling the search as a contingent problem.
Their approach is limited in applicability since it only works on deterministic problems and the features they use are hand-coded.
Aguas et al.
(2016) utilize small example policies to synthesize hierarchical  nite state controllers that can call each other.
However, their approach requires all training data to be provided upfront.
D2L
(Francès et al., 2021) utilizes description logics to automatically generate features and reactive policies based on those features.
Their approach comes with no guarantees for  nding a solution and can only work on deterministic problems.
Deep Learning based approaches such as ASNets (Toyer et al., 2018) learn generalized policies for
SSPs using a neuro-symbolic approach.
They use the action schema from PPDDL to create alternating action and proposition layers.
They do not learn generalized controllers and instead duplicate weights in a post-processing step to represent the generalized policy.
GRL (Karia and Srivastava, 2022) uses
Description Logic based feature evaluations of states as input to a neural network for computing generalized policies in RL settings.
A common limitation of such approaches is the lack of any interpretability as well as theoretical guarantees of completeness or optimality.
Furthermore, these approaches can be susceptible to catastrophic forgetting. GPAs are interpretable, provide strong theoretical guarantees, and do not lose any information when training data is presented as a stream.
Our approach differs from existing work in several aspects.
Our approach constructs a domaindependent GPA automatically without any human intervention.
Using canonical abstraction, we lift problem-speci  c characteristics like object names and object counts in a domain-independent fashion.
Another key difference between other techniques is that our approach can easily incorporate solutions from new examples into the GPA without having to remember any of the earlier examples.
This allows our learning to scale better and can naturally utilize leapfrog learning (Groshev et al., 2018;
Karia and Srivastava, 2021, 2022) when presented with a large problem in the absence of training data.
Finally, our approach comes with guarantees of completeness and hierarchical optimality given the training data presented.
This means that if a solution exists, our approach will  nd it (Thm. 2) and it will be guaranteed to be hierarchically optimal or better (Thm. 3).
Conclusions and Future Work
We show that non-deterministic Generalized Policy Automata (GPAs) constructed using solutions of small example SSPs are able to signi  cantly reduce the computational effort for  nding solutions for larger related SSPs.
Furthermore, for many benchmark problems, the search space pruned by
GPAs does not prune away relevant transitions thus allowing our approach to compute policies of comparable cost in a fraction of the effort.
Our approach comes with guarantees of hierarchical optimality and also comes with the guarantee of always  nding a solution to the SSP.
There are several interesting research directions for future work.
Description Logics are more expressive than canonical abstractions and have been demonstrated to be effective at synthesizing memoryless controllers for deterministic planning problems (Bonet et al., 2019).
Our approach can easily utilize any relational abstraction and it would be interesting to evaluate the ef  cacy of description logics.
Finally, our approach is applicable when solutions have a pattern.
We believe that more intelligent training data generation methods could help improve performance in domains like
Delicate Can.
We plan to investigate these directions of research in future work.
Acknowledgements
We would like to thank Deepak Kala Vasudevan for help with a prototype implementation of the source code.
We would like to thank the Research Computing Group at Arizona State University for providing compute hours for our experiments.
This work was supported in part by the NSF under grants IIS 1909370 and IIS 1942856.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately re  ect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Sec. 4.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] Complete proofs are included in Appendix A.1.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] A link to the public version of the source code has been added to the paper as a footnote.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] The main paper describes signi  cant details of training.
Detailed descriptions of the entire setup are included in Appendix B. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Fig.
shows error bars for the times.
Standard deviations for costs are included in Appendix B as is noted in the caption for Fig.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Sec.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We appropriately cite the problem generators that we used in Sec. 4.
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identi  able information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]