Inductive Synthesis of Finite-State Controllers for POMDPs
Abstract
We present a novel learning framework to obtain finite-state controllers (FSCs) for partially observable Markov decision processes and illustrate its applicability for indefinite-horizon specifications.
Our framework builds on oracle-guided inductive synthesis to explore a design space compactly representing available FSCs.
The inductive synthesis approach consists of two stages:
The outer stage determines the design space, i.e., the set of FSC candidates, while the inner stage efficiently explores the design space.
This framework is easily generalisable and shows promising results when compared to existing approaches.
Experiments indicate that our technique is (i) competitive to stateof-the-art belief-based approaches for indefinitehorizon properties, (ii) yields smaller FSCs than existing methods for several POMDP models, and (iii) naturally treats multi-objective specifications.
INTRODUCTION
Partially observable MDPs (POMDPs) model sequential decision making processes in which the agent only observes limited information about the current state of the system [Smallwood and Sondik, 1973, Kaelbling et al., 1998].
The key challenge in the analysis of POMDPs is to compute a policy satisfying some constraints, captured as a threshold on (discounted) reward or as a task description given in, e g., a temporal logic.
In full generality, policies need arbitrary memory to reflect the belief state of the agent.
Point-based [Pineau et al., 2006, Spaan and Vlassis, 2005] and Monte Carlo methods [Silver and Veness, 2010] excel in finding such policies.
Solving the generally undecidable policy learning problem profits from having complementary approaches in the portfolio.
A natural alternative is to search for (small) finite-state controllers (FSCs)
[Hansen, 1998].
Such controllers provide benefits in terms of explainability [Bonet et al., 2010, Wang and Niepert, 2019], resourceconsumption [Grze ´ s et al., 2013], and generalisability [Inala et al., 2020].
Recently, an automata learning framework has been proposed for synthesising permissive FSCs [Wu et al., 2021].
In this paper, we propose a novel approach— inductive synthesis —to find FSCs for POMDPs.
Inductive synthesis is a technique developed in the context of program synthesis, originally proposed by Church in the 1950’s, the task to construct a program that provably satisfies a given formal specification.
As developing a program (or in this context, a controller) from scratch is mostly infeasible, variants emerged, most notably syntaxguided synthesis
[Alur et al., 2015, 2018] variations such as sketching [Solar-Lezama et al., 2006].
In sketching, the user provides a sketch that outlines a controller implementation, and a specification that constrains the controller’s behaviour.
The principal engine behind many instances of sketching is (oracle-guided) inductive synthesis
[Jha and
Seshia, 2017] and falls in a more general framework of learner-teacher frameworks.
In a nutshell, this methodology suggests to heuristically guess candidate solutions, to validate them, and in case the solution is not satisfactory, learn in order to refine the search heuristic.
The successful application of inductive synthesis has inspired numerous applications beyond classical programming, including recent works on sketching of probabilistic programs [Nori et al., 2015, Ceska et al., 2021, Andriushchenko et al., 2021b] and (variations of) programmatic reinforcement learning [Verma et al., 2018, Inala et al., 2020].
This paper proposes inductive synthesis to search for FSCs in POMDPs.
Our inductive synthesis framework works in two stages, see
Fig.
Let us first discuss the outer stage.
Here, a learner constructs a design space containing (finitely many) FSCs.
A teacher provides the ‘best’ FSC within this design space, and potentially additional diagnostic information.
The learner either accepts the FSC provided by the teacher as final result, or adapts the design space.
Naturally, teachers will provide much better FSCs much faster whenever the design space
[Figure]
for these FSCs is small.
The key ingredient for the outer stage is thus to start with a small design space and to strategically modify this design space based on the obtained feedback from the teacher.
A similar idea was proposed in [Kumar and Zilberstein, 2015], where the entropy of the observations is used as criterion for adding memory to the
FSC.
We use the FSC returned by the teacher together with the state-values induced by this FSC.
Additionally, we use an abstraction oracle, see below.
The inner stage describes the internals of the teacher that determines the ‘best’ FSC within the design space.
The teacher may naively use enumeration, but can also be realised using branch-and-bound
[Grze ´ s et al., 2013] or mixed-integer linear programming (MILP) [Amato et al., 2010, Kumar and Zilberstein, 2015].
We realise the teacher by (another) inductive synthesis loop.
We search for an FSC by symbolically representing the design space as a propositional logic formula.
The policy evaluation analyses a fixed policy w.r.t.
the given specification (e.g., a reward function and a threshold).
If the policy refutes the specification, the evaluation engine indicates the distance to satisfaction (e.g., the achieved value) as well as conflicts—critical parts of the
FSC that suffice to violate the specification—that are used to prune the search design space [Ceska et al., 2021].
Both learning stages have access to an additional oracle that, inspired by Andriushchenko et al.
[2021a], overapproximates the design space.
This larger abstract design space can efficiently be analysed as the underlying problem solved by the abstraction oracle resembles the analysis of fully observable policies.
The oracle yields constraints to what the best FSC within the original design space will possibly achieve.
This information is an essential ingredient to guide the search in both stages.
The separate policy evaluation—a natural component in an inductive synthesis framework—brings some advantages.
The policy evaluation (i.e., solving systems of linear equations) via dedicated algorithms is faster than letting an (MI)LP solver solve these equations [Dehnert et al., 2014].
This improves upon performance of MILP-based approaches (either primal [Winterer et al., 2020] or dual [Kumar and Zilberstein, 2015]) for FSC synthesis.
Furthermore, as the policy is fixed, our framework provides an elegant alternative to existing approaches for constrained
POMDPs [Poupart et al., 2015, Khonji et al., 2019] and multi-objective POMDPs [Soh and Demiris, 2011, Roijers et al., 2013, Wray and Zilberstein, 2015].
It additionally paves the way to learn robust FSCs for POMDPs with imprecise probabilities, similar to [Cubuktepe et al., 2021].
We instantiate our framework to learn deterministic FSCs, i.e., FSCs that do not use randomisation.
Finding optimal deterministic FSCs is NP-complete whereas finding optimal randomised FSCs is ETR-complete 1 [Junges et al., 2018, 2021].
Algorithmically, finding randomised FSCs requires solving non-convex optimisation problems with thousands of variables.
This often limits the guarantees on global (almost-)optimality that are practically feasible [Kumar and
Zilberstein, 2015].
Deterministic FSCs are additionally beneficial in terms of reproducibility of their behaviour, which is useful for debugging.
We use an evaluation framework that supports indefinite horizon queries, e.g., queries with a discount factor one.
These queries generalize infinite horizon properties with a discount factor < 1 and finite-horizon settings as used in Goal-POMDPs [Bonet and Geffner, 2009,
Kolobov et al., 2011].
These queries naturally occur when using temporal logic specifications and are particularly adequate for safety-critical aspects.
The experimental evaluation shows the applicability of our approach on a wide range of benchmarks with promising results.
Particularly, it significantly outperforms approaches based on MILP optimisation.
We further compare it with the state-of-the-art belief-based approaches, namely, with recent works in formal verification on under-approximation for indefinite-horizon specifications [Norman et al., 2017, Bork et al., 2022].
Our inductive synthesis approach is highly competitive and for several POMDPs (having a moderate number of observations/actions and large/infinite belief-space), it is able to find small FSCs improving lower bounds of existing solutions.
PROBLEM STATEMENT
A (discrete) distribution over a finite set
X is a function µ : X → [0 , 1] s.t.
(cid:80) x µ ( x ) = 1 .
The set Distr ( X ) contains all distributions over X .
A Markov decision process (MDP) is a tuple M =
( S, s 0 , Act, P ) with a finite set S of states , an initial state s 0 ∈ S , a finite set Act of actions , and a transition probability function P ( s ′ | s, a ) that gives the probability of evolving to s ′ after taking action a in s .
A Markov chain (MC) is an
MDP with | Act | = 1 ; its transition function is written as
P ( s ′ | s ) .
MDPs can additionally be equipped with a reward function r ( s, a ) .
We do not use discount factors, see the paragraph on specifications below.
T 0/ 1/ 2/ 3/ 4/ 5/ 6/ 7/ 8/ 9/
A Partially Observable MDP (POMDP) M
=
( S, s 0 , Act, P, Z, O ) extends MDP M with a finite set Z of observations , and a (deterministic) observation function 2 O that returns for every state s an observation
O ( s ) =
z ∈ Z .
The observation z ∈ Z is said to be trivial if there is only one state s ∈ S with O ( s ) =
z.
Finite State Controllers (FSCs) are automata that compactly represent policies.
We call its states (memory) nodes to distinguish them from POMDP states.
We also refer to an
FSC with k nodes as having k memory.
FSCs in the literature come in various styles, in particular either as Moore machines, with the output—the action it selects—determined by the node, or as Mealy machines, with the output determined by the taken transition [Amato et al., 2010].
In the context of sketching FSCs and their inductive exploration, it is convenient to describe FSCs as Mealy machines.
Furthermore, we restrict ourselves to deterministic FSCs.
Formally, a finite-state controller (FSC) for a POMDP M is a tuple F = ( N, n 0 , γ, δ ) , where N is a finite set of nodes , n 0 ∈ N is the initial node , γ ( n, z ) determines the action when the agent is in node n and observes z , while δ updates the memory node to δ ( n, z ) , when being in n and observing z .
For | N | =
k , we call an FSC a k -FSC.
Imposing k -FSC F onto POMDP M yields the induced Markov chain M F =
( S F , ( s 0 , n 0 ) , P F ) with
S F = S × N
and using 3 z
= O ( s ) :
P F (( s ′ , n ′ )
| ( s, n )) =
P ( s ′ | s, γ ( n, z )) ·
[ δ ( n, z ) = n ′ ] .
Example 1.
As running example, we use a simple variant of the maze problem [Hauskrecht, 1997], where an agent tries to reach the state s T , modelled by the POMDP
M with S =
{ s 0 , . . .
s 9 , s T } , Act = { u, d, l, r } , and
Z = { z 0 , . . .
, z 5 } .
The initial state is given by a uniform distribution over S .
Fig.
2 (left) depicts P and O where state s x is labelled by x/y with x the state index and y its observation, i.e., O ( s x ) =
z y .
The arrow direction from x/y to x ′ /y
′ represents the action; e.g., → corresponds to action r .
The maze is slippery.
An action is successful with probability 0 .
9 ; with 0 . 1 , the agent does not move.
Actions without effect are omitted from the figure.
Fig.
2 (right, top) illustrates a fragment of a 2-FSC where γ ( n 0 , z 0 )
=
γ ( n 0 , z 1 ) =
r (for memory node n 0 and observations zand z 1 , action r is chosen), γ ( n 0 , z 3 )
=
γ ( n 1 , z 1 ) =
l , δ ( n 0 , z 0 )
=
δ ( n 0 , z 1 ) = n 0 (memory node n 0 is not changed for z 0 an z 1 ) and δ ( n 0 , z 3 ) =
δ ( n 1 , z 1 ) = n 1 .
This
FSC tries to resolve the inconsistency (formalised later) in the observation z 1 , i.e., in s 1 the action r is optimal but in saction l is optimal (w.r.t.
reaching s T ).
Fig.
2 (right, bottom) illustrates a fragment of the induced MC containing two copies of s 2 .
Specifications contain two parts: a set of constraints given by quantitative properties and a single optimisation objective .
Constraints are defined as indefinite-horizon reachability and expected reward properties, but our approach also supports more general probabilistic temporal logic properties [Baier and Katoen, 2008] 4 Let target set T ⊆ S , thresholds λ 1 ∈ [0 , 1] and λ 2 ∈ R + and ▷◁ ∈ {≤ , ≥} .
The
POMDP M under FSC F satisfies the constraint P ▷◁λ 1 if the probability Pr F of reaching T in the induced MC M F meets ▷◁ λ 1 .
Similarly, the constraint R ▷◁λ 2 is satisfied if the expected reward R F accumulated in MC M F until reaching T meets ▷◁ λ 2 .
We call an FSC F admissible (for
M ), if M under F satisfies the given (set of) constraint(s).
Objectives either minimise or maximise reachability probabilities (as in goal-POMDPs) or (un)discounted expected reward properties, denoted as P ∗ and R ∗ respectively for ∗ ∈ { min , max } .
The probability or reward obtained by
FSC F on M is called the value of F .
For conciseness, we assume throughout the paper that the specification contains a maximisation objective.
Minimisation is analogously supported (but may require flipping bounds and inequalities).
Problem statement.
We aim to construct an algorithm that: i) quickly finds a (small) admissible FSC F and ii) incrementally improves F w.r.t.
the optimisation objective.
We can view the algorithm as solving a sequence of decision problems, where the first decision problem is to find some admissible FSC F 0 and decision problem i +1 is to find an admissible FSC F i +1 whose value improves upon the value of the previous FSC F i .
INDUCTIVE EXPLORATION OF FSCS
This section presents the inner loop (see Fig. 1) in which we search among a given set of k -FSCs.
Before we describe the ingredients, we formalise the representation of the set of k -FSCs.
We then outline the two oracles that our search can use to prune the search space.
A hybrid strategy [Andriushchenko et al., 2021a] combines the two oracles by switching based on perceived performance while communication between the oracles takes place.
3.1 FAMILIES OF FSCS
A POMDP and a single FSC yield a single induced MC.
A
POMDP and a set of FSCs thus induces a set of MCs.
The set of FSCs has additional structure which enables concisely describing the set of MCs.
We first consider full FSCs where for each observation the same amount of memory is used and where there are no restrictions on the memory updates.
We generalise this to a class of reduced FSCs that are more memory efficient.
Definition 1.
A family of full k -FSCs is a tuple F k = ( N, n 0 , K ) , where N is a set consisting of k nodes, n 0 ∈ N is the initial node,
K = N × Z is a finite set of parameters each with domain V ( n,z ) ⊆
Act × N .
From a family, one may obtain a k -FSC by choosing values for each parameter, effectively determining the action γ ( n, z ) and the next node δ ( n, z ) .
Thus, each family describes a set of FSCs by varying the substitutions of the parameters.
We often use F k to denote such a set of k -FSCs.
We remark that this set contains O (( | Act || N | ) ( | N || Z | ) )
many
FSCs.
A POMDP M and a family F k naturally induces the family of MCs M F k = {M F | F ∈ F k } .
Example 2.
The family F 2 of all 2-FSCs for our maze problem is given by N = { n 0 , n 1 } , K = { ( n i , z j ) | i ∈ { 0 , 1 } ∧ j ∈ { 0 , . . .
, 5 }} , and V ( n,z )
= { u, d, l, r } × { n 0 , n 1 } for all ( n, z ) ∈ K .
While FSCs have k available memory nodes in conjunction with every observation, memory may only be required in some observation (see e.g., the running example).
Therefore, we consider reduced FSCs given by a memory restriction µ : Z → N , where µ ( z ) determines the number of memory nodes used in the observation z .
Definition 2.
A reduced family F µ given by the memory model µ is a sub-family of F k for k = max z ∈ Z { µ ( z ) } where ( n, z ) ∈ K implies n ≤ µ ( z ) , and the domains
V ( n,z ) are as in F k .
If a memory update δ ( n, z ) = n ′ is invalid in the resulting observation z ′
(i.e., n ′ > µ ( z ′ ) ), then δ ( n, z ) = n 0 .
The reduced family for k -FSCs yields a significantly reduced number of parameters (cid:80)
z ∈ Z { µ ( z ) } .
This is less than k · | Z | if µ ( z ) <
k for most observations z .
Indeed, in many experiments, we can use µ ( z ) = 1 for all but a few observations.
Such reduction has several key benefits:
Foremost, the family of reduced FSCs induces a smaller design space, but it also yields FSCs where less memory is needed, which can be beneficial for their interpretability.
3.2 MDP-ABSTRACTION TEACHER
We introduce a teacher which, for a POMDP and a FSCfamily, provides safe upper and lower bounds on the value of the FSCs in this family.
Instead of considering the individual
FSCs, the oracle considesr an abstraction (represented as a single MDP) of the set of induced MCs.
Definition 3.
MDP A F = ( S × N, ( s 0 , n 0 ) , Act F , P F ) is an abstraction of MC family M F with Act F = Act × N and P F (( s ′ , n ′ )
| ( s, n ) , ( a, n ′ )) =
P ( s ′ | s, a ) if ( a, n ′ ) ∈
V ( n,O ( s )) , and 0 otherwise.
For MDPs and our specifications, it suffices to consider deterministic memoryless policies, i.e., policy π for MDP A F is a function π : S × N → Act F .
It is consistent (w.r.t.
the observations) if O ( s ) =
O ( s ′ ) implies π (( s, n ))
=
π (( s ′ , n )) for all s, s ′ ∈ S, n ∈ N .
The set of consistent policies in A F corresponds to the policies for the family
F .
The policy π is inconsistent in a FSC-family parameter ( n, z ) ∈ K if ∃ s, s ′ ∈ S :
O ( s ) =
O ( s ′ )
=
z ∧ π (( s, n ))
̸
= π (( s ′ , n )) .
It is inconsistent in observation z ∈ Z , if it is inconsistent in the parameter ( n, z ) ∈ K for some n ∈ N .
Example 3.
Assume we want to maximise the probability to reach s T .
The stars in Fig.
2 represent the optimal policy π ∗ in MDP A F where F is set of all 1-FSCs for the maze problem.
π ∗ is inconsistent in the observations z 1 and z 4 .
The analysis of MDP A F provides useful information about the family F .
For conciseness, consider the constraint P ≥ λ bounding the reachability probability to states in T .
We can compute the policy π ∗ in A F that maximises this reachability probability.
In particular, this policy achieves probability
Pr π ∗ .
If Pr π ∗ < λ , then all F ∈ F violate the constraint
P ≥ λ and F can be safely discarded.
Otherwise, we check the consistency of policy π ∗ .
If π ∗ is consistent, it represents an FSC satisfying P ≥ λ .
Similarly, a minimising policy may witness that the entire family F satisfies P ≥ λ .
If analysing
A F is inconclusive, we refine F by splitting, see below.
The optimisation objective is handled by iteratively updating a new (initially trivial) constraint representing the running value of the optimum so far.
Once an admissible policy π is found, we update the new constraint according to the objective value that π achieves.
Reasoning about multiple constraints works as follows.
If the entire family F violates some constraint, F is discarded.
If F satisfies the constraint, this constraint will neither be checked again for F nor for its subfamilies.
Otherwise, if the analysis of F was inconclusive with respect to some constraint, F is refined.
Beyond pruning families, analysing A F provides statevectors ub and lb such that ∀ s ∈ S , lb ( s ) and ub ( s ) bound the probability to reach T from s .
These bounds are used in the inner and outer synthesis loop, as we will see below.
General refinement strategy
The refinement strategy is a key component in driving the exploration of the family F .
It decomposes F into sub-families by splitting the domain of selected parameters from K .
In contrast to the general strategy used in program synthesis [Ceska et al., 2019], we leverage the specific topology of the FSC families.
The key idea is to examine the inconsistencies of the policy π ∗ obtained for a given constraint.
Assume π ∗ is a maximising policy inconsistent in parameter ( n, z ) ∈ K .
We estimate the significance of this inconsistency as the average variance (with respect to inconsistent actions) of ub (( n, s )) with
O ( s ) =
z , where ub (( n, s )) is weighted by the expected number of visits of the state ( n, s ) in the MC induced by π ∗ (if π ∗ is minimising, we use lb ).
We refine F using the most significant inconsistent parameter ( n, z ) .
Assume it has domain V ( n,z ) and π ∗ selected options v 1 , . .
. , v i .
We partition
V ( n,z ) into { v 1 } , . .
. , { v i } and V ( n,z ) \{ v 1 , . . . , v i } , and create i +1 corresponding subfamilies.
This removes the inconsistency of ( n, z ) by considering the selected options v 1 , . .
. , v i within different sub-families.
Incomplete refinement strategy We suggest the following incomplete refinement strategy to focus the search (at the cost of completeness).
In particular, we restrict the exploration to FSCs that are structurally close to π ∗ as follows:
we i) fix the options selected by π ∗ in perfectly observable states, ii) fix the options in the consistent parameters, and iii) remove options in the inconsistent parameters that were not selected by π ∗ , i.e., the set V ( n,z ) \{ v 1 , . .
. , v i } .
3.3 COUNTEREXAMPLE-BASED TEACHER
The MDP abstraction employs deductive reasoning: It talks about a set of FSCs at once to deduce conclusions about the individual members of this set.
In this subsection, we discuss the orthogonal, inductive, approach.
We suggest a (smart) enumeration over individual FSCs inspired by Ceska et al.
If the FSC is satisfactory, i.e., it is admissible and has good value, this helps the teacher.
Otherwise, if the FSC is not satisfactory, we learn facts, called counterexamples , that help us to avoid considering other FSCs.
To realise this teacher, we represent the FSCs that have not been pruned as a propositional formula 6 .
We use the SMT solver CVC5 [Barbosa et al., 2022] (over quantifier-free bounded integers) to effectively manipulate the propositional formula and to find FSCs that have not been pruned.
We assume a constraint P ≥ λ
( ♢ T ) , a family F , the state- vector ub obtained from the maximising policy π ∗ in MDP
A F as discussed in Sec. 3.2, and an FSC F ∈ F .
Definition 4.
A counterexample (CE) for FSC F and P ≥ λ is a subset C ⊆ S F that induces the sub-MC of M F given as M FC =
( C ∪ succ( C ) ∪ { s ⊥ , s ⊤ } , ( s 0 , n 0 ) , P ′ ) with P ′ ( s ) =
 
P F ( s ) if s ∈ C, [ s ⊤ (cid:55)→ ub ( s ) , s ⊥
(cid:55)→ 1 − ub ( s )] if s ∈ succ( C ) \ C, [ s (cid:55)→ 1] if s ∈ { s ⊤ , s ⊥ } , where succ( C ) is the set of direct successors of C , and the probability to reach T ∪ { s ⊤ } in M FC is < λ .
Intuitively, in the sub-MC, states s outside the CE C evolve to s ⊤ with probability ub ( s ) , the maximal probability to reach T from s in the family F (i.e., the worst-case possible in F ).
They evolve to s ⊥ with probability 1 − ub ( s ) , the minimal probability to avoid T in F .
For ( s, n ) ∈ C , the parameter ( n, O ( s )) ∈ K is called relevant .
The CE for the constraint P ≤ λ is defined similarly using lb rather than ub .
For each F ′ ∈ F that for each relevant parameter in a CE C uses the same values as F , it holds that P F ′ < λ .
Therefore, we can safely remove F ′ from the design space.
We say that
C generalised to the set of all such F ′ .
Smaller CEs lead to generalisation to larger families of FSCs.
As computing minimal CEs is NP-complete [Funke et al., 2020], we adopt the greedy approach from [Andriushchenko et al., 2021a].
Handling multiple constraints is straightforward as we can compute the CE for each constraint violated by the FSC F .
This can potentially improve the pruning.
Similarly as the incomplete refinement strategy in Sec. 3.2, we consider an incomplete generalisation of the CEs.
In particular, we redefine the notation of relevant parameters.
The parameter ( n, O ( s )) for ( s, n ) ∈ C is relevant only if the observation O ( s ) is inconsistent in A F or the option selected by F is different from the options selected by π ∗ .
This leads to more aggressive pruning and restricts the exploration to the FSCs that are topologically close to π ∗ .
Example 4.
Consider a variant of our maze problem with initial state s 0 , family F of all 1-FSCs, where the available set of actions in the observation o 3 is restricted
s.t. V ( n 0 ,z 3 ) ∈ { u, d, r } × { 0 } .
Let FSC F as in
Fig 3 (left).
The right part illustrates the induced MC and the middle part shows the CE C for the constraint P ≥ 1 .
Note
P (( s 4 , n 0 ) , s ⊥ )
= 1 as ub ( s 4 ) = 0 .
Thus, the relevant parameters are ( n 0 , z i ) for i ∈ { 0 , 1 , 2 } .
The generalisation of
C enables pruning a significant part of F .
Under incomplete generalisation, the parameter ( n 0 , z 0 ) is not relevant as it is consistent in A F and F picks the same option as π ∗ .
X/r
Y/d 0, 1, 2, 4,0 3, 5,0 0, 1, 2,
[Figure]
MEMORY INJECTION STRATEGY
This section discusses the outer stage of our approach, cf.
Fig.
1, in which the learner decides where to search.
In particular, a subset of FSCs is selected and passed onto the teacher, as outlined in Sec.
We assume access to an abstraction oracle that yields bounds on the value for every state based on an abstraction scheme outlined in Sec. 3.2.
The learner processes this information and derives a new design space.
It does so by combining three ingredients: 1.
Adding memory :
By allowing FSCs to store more information, we (drastically) increase the design space.
We allow to locally increase memory to keep the growth manageable.
Removing symmetries : Similar to [Grze´s et al., 2013], it is unnecessary to include symmetric FSCs in the design space.
Analysing abstractions : Use the results from the abstraction to guide the search.
4.1 ADDING MEMORY
The key idea of the memory injection strategy is to use the diagnostic information obtained from the preceding inner loop exploring the design space represented by a family
F to construct a new family F ′ , say.
These families are based on two fixed memory structures (either as a full or reduced FSC).
On constructing F ′ , memory can be added that corresponds to one of the observations, see Sec. 3.1.
This section outlines where to add the memory.
To decide how to extend the family F , we use the following information: 1.
The maximising policy π ∗ in MDP A F together with its corresponding bounds ub 7 . 2.
The FSC F ∗ in F obtained from the teacher.
That FSC is not available if the FSC is inadmissible or if the teacher is aborted.
If only π ∗ is available, the memory injection strategy employs a similar idea as the refinement strategy described in Sec. 3.3.
In particular, it evaluates the significance of the inconsistent observations w.r.t.
π ∗ by aggregating the significance of their inconsistent parameters as described above.
By adding the memory to the most significant inconsistent observation, we try to resolve the inconsistency and drive the search towards an FSC that mimics π ∗ as often as possible, i.e., along most paths.
If additionally F ∗ is available, the idea is similar: to obtain a better FSC in the next iteration, we identify the observation in which the choices of F ∗ differ from the observation-free policy π ∗ the most.
That is, the inconsistency measure χ ( s ) for state s is now the absolute difference in ub ( s ) (for the maximising property) w.r.t.
two actions: the one provided by π ∗ and the one from F ∗ .
The inconsistency measure of the observation z is now the weighted average of χ ( s ) across all states s with O ( s )
=
z , where the weight for state s is the expected number of visits of s in the FSC F ∗ .
Note that in both cases the proposed inconsistency measure is just a heuristic and neither guarantees that adding memory to the selected observation will improve the value of the
FSC, nor that the memory injection strategy will eventually find the optimal F ∗ .
In fact, without additional modifications to the strategy, the heuristic will keep adding memory to a single observation because adding memory does not resolve the inconsistencies.
To mitigate this problem, it is crucial to employ the symmetry reduction.
4.2 SYMMETRY REDUCTION
Adding memory to a selected observation typically creates a family F that includes FSCs having the same value due to certain symmetries in their topology.
Removing these symmetries from F reduces: i) the size of F and ii) the number of inconsistencies of the policy π ∗ obtained from
A F , see below.
As a result of resolving these inconsistencies, the memory injection strategy is capable of better recognising where else in the POMDP memory is needed.
The importance of symmetry breaking has been recognised by Grze ´ s et al.
[2013], who proposed a generation strategy for isomorphism-free Moore automata.
We propose a different approach based on restricting the family of candidate
FSCs.
We illustrate where the symmetries are introduced as well as how to deal with them in the example below.
Example 5.
Consider again the maze problem from Example 1 and the specification to minimise the expected number of steps to reach T .
Note that this includes an implicit constraint to reach T .
Let F 1 be the family of all 1-FSCs (no memory was added).
The inner loop detects that there is no admissible 1-FSC satisfying the constraint: in observation zwe need to be able to pick both actions r and l , and similarly for observation z 4 .
Assume that the minimising policy π ∗in A F 1 reveals that the most significant inconsistency is the one in observation z 1 .
Adding memory to z 1 has a twofold effect on the resulting design space F 2 .
First, a new parameter ( n 1 , z 1 ) is introduced that encodes action selection in newly created copies of states with observation z 1 .
Second, each state having successor s with observation z 1 must be able to choose whether to go to ( s, n 0 ) or its copy ( s, n 1 ) .
In our case, the domains of parameters ( n 0 , z 0 ) , ( n 0 , z 2 ) , ( n 0 , z 3 ) as well as of parameters ( n 0 , z 1 ) and ( n 1 , z 1 ) (remember the self-loops) will now be
{ u, r, d, l } × { n 0 , n 1 } .
Before proceeding with the inner loop, we first remove symmetric assignments from the family F 2 .
That is, consider FSC ( γ, δ ) with γ ( n 0 , z 1 )
=
l and γ ( n 1 , z 1 )
=
r .
Clearly, this FSC achieves the same value as the symmetric FSC ( γ ′ , δ ′ ) with γ ′ ( n 0 , z 1 ) =
r , γ ′ ( n 1 , z 1 )
=
l and δ ′ ( · , z ) = n 1 if δ ( · , z ) = n 0 and vice versa, for each predecessor observation z of z 1 .
The family F ′ 2 should include only one of the two FSCs, so we modify the domains of parameters ( · , z 1 ) as follows: V ( n 0 ,z 1 )
= { u, d, r } × { n 0 , n 1 } and V ( n 1 ,z 1 ) =
{ u, d, l } × { n 0 , n 1 } .
The inner loop again detects that no admissible solution exists.
Minimising policy π ∗ 2 now contains a single inconsistency in observation z 4 , which we amend by adding memory to z 4 .
In the resulting family F 3 , we introduce parameter ( n 1 , z 4 ) for action/memory selection in the new copy, and modify parameters ( n 0 , z 5 ) , ( n 0 , z 4 ) and ( n 1 , z 4 ) to enable the transition to the newly created copies (other successors of observation z 4 already account for both possible memory updates).
As in the previous case, we break the symmetry in action selection in ( · , z 4 ) :
V ( n 0 ,z 4 ) = { u, r, l } × { n 0 , n 1 } and V ( n 1 ,z 1 ) =
{ d, r, l } × { n 0 , n 1 } .
The third iteration of the inner loop finally yields an optimal FSC with value 7 . 16 .
No additional memory injection can improve upon this.
More generally, for each observation z , we keep a list I zAct of actions in which parameters ( · , z ) were inconsistent.
This list is updated each time we update memory for observation z .
Upon adding memory to z , we apply the symmetry reduction to the corresponding domains of parameters ( n i , z ) .
For simplicity, let |I zAct | = µ ( z ) , as in Example 5.
In such a case, V ( n i ,z ) is set to ( Act ( z ) \I zAct ∪ {I zAct [ i ] } ) × N , where I zAct [ i ] is the i -th action in the list I zAct .
This ensures that for each action a ∈ I zAct there is exactly one parameter
V ( · ,z ) where a is available.
If |I zAct | ̸ = µ ( z ) , the construction of domains V ( · ,z ) becomes more involved: one must take possible inconsistencies in memory updates into account.
For a comprehensive description of the symmetry reduction, we refer to the implementation (see Sec. 5).
In general, there are two extreme ways one can proceed with the symmetry reduction.
Either one can enable all of the actions/memory updates in V ( · ,z ) , which will undermine the observation selection discussed in Sec. 4.1.
The other extreme is to disable choices in V ( · ,z ) arbitrarily.
This may lead to incompleteness, as the following example shows.
Example 6.
Assume the following modification of Example 1, where an agent now experiences a slight drift to the west: upon choosing the direction, the agent will move to the selected direction with probability 0.9 and will otherwise move one cell to the left (if available).
For instance, when moving down from the state 2, the agent might instead end up in the state 1.
FSC synthesis proceeds similarly as in Example 5, so assume that after two memory injections and symmetry reductions we end up with the same family F ′ 2 having V ( n 0 ,z 1 )
= { u, d, r } ×
( n 0 ,z 4 ) = { u, r, l }×{ 0 , 1 } , V ( n 1 ,z 1 ) = { d, r, l }×{ 0 , 1 } .
The optimal assignment for the state 2 is γ ( n 0 , z 2 ) =
d (moving down) and δ ( n 0 , z 2 ) = n 1 , since ( n 1 , z 4 ) is the only parameter that enables movement down from the state 6.
However, if due to drift the agent ends up in state ( s 1 , n 1 ) , it cannot move right ( r ̸∈ V ( n 1 ,z 1 ) ) and is forced to move sub-optimally.
In our case, we could have preserved the optimal solution if we had switched the order of e.g. second symmetry reduction, although predicting a non-conflicting reduction order would require additional analysis.
EXPERIMENTAL EVALUATION
Our evaluation focuses on the following questions:
Q1:
How does our approach compare with state-of-the-art belief-based approaches?
Belief-based approaches are the widespread approach to solving POMDPs.
They implicitly or explicitly approximate the large or infinite belief-MDP instead of searching for policies with a particular structure.
We compare with the approaches by Norman et al.
[2017] (implemented in PRISM Kwiatkowska et al.
[2011]) and by
Bork et al.
[2022] (implemented in Storm [Dehnert et al., 2017]).
These methods provide state-of-the-art techniques for finding policies in belief MDPs for indefinite-horizon specifications, i.e., without discounting.
Q2:
How does our approach compare to state-of-the-art approaches to synthesise deterministic FSCs?
To this end, we qualitatively compare with the state-of-the-art dual MILP formulation from [Kumar and Zilberstein, 2015] which uses a max-entropy strategy for adding memory nodes.
We also consider a recent alternative formulation of a primal MILP in [Winterer et al., 2020] for multi-objective specifications.
Q3:
What is the effect of our heuristics on the run-time and the value of the resulting FSCs?
We discuss additional insights from an ablation study in which we discuss which settings yield the best performance.
Selected benchmarks and setup The framework outlined above has been implemented in PAYNT Andriushchenko et al.
[2021b], a tool for inductive synthesis of probabilistic programs 8 .
Unless mentioned otherwise, we used benchmarks from [Bork et al., 2020, 2022] extended by a few more involved variants.
Table 1 lists the statistics of the models including the number of states, the overall number of actions, and the number of observations.
Our experiments run on a single core of a machine equipped with an Intel i5-12600KF @4.9GHz CPU and 32 GB of RAM.
An artefact allowing one to reproduce our experimental evaluation is available at https://doi.org/10.5281/zenodo.6637489 .
Threads to validity This evaluation focuses on showing the potential of our approach using benchmarks from synthesis the verification literature.
While the comparison with implementations in Storm and PRISM is thorough, other algorithms were not available and thus we resort to comparing with the performance reported in those papers.
There is a dire need for a more structural comparison over different algorithms.
Furthermore, the experiments suffice to provide insights in the performance of the algorithm, but not to draw conclusions about the relative relevance of individual heuristics.
Finally, while our approach is general and could be applied to queries such as expected discounted rewards, it can only be competitive if it is tailored to that setting.
Q1: Comparison to belief-based methods
Table 1 summarises key experimental results related to Q1 .
The columns list the following information (from left to right): the model and its variant, the model statistics, the bounds provided by [Norman et al., 2017] and its run-time, the lower bounds provided by [Bork et al., 2022] and its run-time (for two settings: the fastest synthesis and the best bound), the results provided by our approach (including the number of added memory nodes) and its run-time (the first interesting solution and the best solution found), and the upper bounds provided by Bork et al.
[2020] allowing us to judge the quality of the lower bounds.
To simplify the presentation, this table shows results achieved by our approach using the default setting (different settings are used for the entries denoted by ∗ ): the inner loop is instantiated by the pure MDP abstraction oracle with the incomplete refinement strategy, and the outer loop uses the memory injection strategy with symmetry reduction.
The impact of our optimisation heuristics as well as the results for multi-objective specifications are discussed under Q3 .
The results demonstrate that our inductive approach is competitive with the belief-state space approximation for indefinite-horizon specifications .
For models with a moderate number of observation/actions, we provide better tradeoffs between the run-time and the values of the found policies.
Moreover, we found small FSCs that improve the lower-bounds in [Bork et al., 2022].
For models with a large number of observations/actions, we found small highvalue FSCs in comparable run-time.
For the Rocks model and a larger Netw model, we failed to find a good solution.
We highlight two interesting results: For Grid-av 4-0 , our strategy injected four memory nodes (see † ) and achieved the bound provided by the observation-free MDP abstraction which guarantees the global optimum.
For Drone 4-2 , we found a very small FSC that achieves the known upper bound on the solution value [Bork et al., 2020].
Q2: Comparison to MILP-based FSC synthesis
A direct comparison with MILP-based approaches is complicated due to limited availability of standardised implementations.
Qualitative comparisons are furthermore complicated by differences in benchmarks.
However, based on the Hallway model from [Kumar and
Zilberstein, 2015] and manually translating it to mimic the effect of discounting and stochastic rewards to an almost equivalent model 9 , we make the following preliminary observation.
The dual MILP optimisation for the fixed-size reactive FSC (equivalent to our 1-FSC) achieved the value 0.32 in 15 minutes.
Given the discount factor 0.95, this corresponds to an FSC where the expected number of steps to reach the target equal to log 0 .
95 (0 . 32) = 22 . 2 .
Using the memory injection strategy, they found an FSC withadditional memory nodes in ∼ 1 hour achieving value 0.46, i.e., 15.1 expected steps.
Our complete strategy explored all 1-FSCs in less than 1s and found a solution achieving 18.
expected steps.
The restricted exploration of full 4-FSCs found a solution achieving 14.9 expected steps in 218s.
The default strategy used in Table 1 (see above) added one memory node and found a solution achieving 16.1 expected steps in less than 1s.
Despite all the limitations of this experimental setup, these results indicate that our approach can be at least as successful as MILP-based synthesis methods.
Similarly, we compare with [Winterer et al., 2020] on the
Grid-av 4-0 model with a constraint on the reachability probability and the minimisation of an reward.
The best solution of the MILP optimisation with a restricted randomisation and memory injection has value 3.43 (found within seconds).
This solution is obtained by our default strategy within 1s by adding one memory node.
In 21s, it added seven memory nodes and found a better solution having value 3.29.
This shows that our inductive approach outperforms the MILP optimisation also on multi-objective specifications .
Q3: The effect of optimisation heuristics
Efficacious heuristics: We generally remark that the design spaces in this paper are several orders of magnitude bigger than the design spaces supported by the more generalpurpose inductive synthesis framework in [Andriushchenko et al., 2021b].
The superior performance can mostly be explained by the tailored representation of the design space and novel search heuristics.
Incomplete search: To find a good policy, it is not necessary to be complete.
The (default) incomplete refinement strategy and CE generalisation is beneficial for handling large number of observations/actions.
The complete exploration fails to find a good solution for, e.g., the Drone models.
In our experiments, we did not observe that the incomplete exploration discards important solutions except the Grid-av 4-0. model.
For that model, the incomplete strategy performsmemory injections and finds in 840s a solution with value 0.92.
The complete strategy performs 8 memory injections
[Table]
and finds in 1189s a solution with value 0.93.
For this benchmark, the best synthesis result reported in Table 1 (column best relied on complete exploration of
Memory injection: this prevents the blowup that just increasing memory nodes has.
Without symmetry reduction, the abstraction-based framework has trouble guiding the search.
Symmetry reduction thus not only reduces the design space but it also guides the memory injection strategy correctly select the most promising observation.
For example, in the
Maze sl model, the memory injection without symmetry reduction repeatedly adds memory to a single observation and the optimal solution is not found.
On the other hand, symmetry reduction can discard an optimal solution as demonstrated on the Grid-av 4-0.1, Maze sl, Hallway and Refuelmodels.
For these models, the column best of Table 1 lists the results of the exploration of full k -FSCs denoted as k f for k ∈ { 2 , 3 , 4 , 5 } .
Hybrid teacher: For the models in Table 1, the use of CEs or a hybrid teacher (combining MDP abstraction and CE pruning) does not improve the synthesis process.
We believe that future work towards CE-guided inductive synthesis may change this balance.
As of now, only for models where the
MDP abstraction is significantly larger than the induced
MCs corresponding to the candidate FSCs, the hybrid approach is superior: e.g., for the Grid 30-sl model, a larger variant of the grid-like model, the MDP abstraction is 15x larger than the individual induced MCs.
With default settings no admissible FSC is found within 30 minutes.
A hybrid teacher helps finding an FSC within 150s that improves the solution found by the belief-based method.
Multi-objective (MO) specifications:
Apart from the MO variant of the Grid-av 4-0 model discussed in Q2 , we also considered a MO variant of the Maze sl model including a more complicated specification with an additional reachavoid constraint.
The constraint restricts the optimal FSC, but the run-time of the synthesis remains < 1s.
CONCLUSION
This paper presents a first inductive-synthesis based framework for finding finite-state controllers (FSCs) in POMDPs.
Key ingredients are the novel heuristics to incrementally construct the memory structure of the FSC as well as two oracles for searching and evaluating families of FSCs.
The experimental results show promising results indicating that this framework is competitive with the state-of-the-art alternatives.
Future work includes the integration of belief-based approaches as an additional oracle.
Acknowledgements
This work has been supported by the Czech Science Foundation grant GJ20-02328Y and the ERC AdG Grant(FRAPPANT).
The authors thank Alexander Bork and Filip
Macák for their support in running the experiments.