Policy Optimization with Linear Temporal Logic Constraints
Abstract
We study the problem of policy optimization (PO) with linear temporal logic (LTL) constraints.
The language of LTL allows flexible description of tasks that may be unnatural to encode as a scalar cost function.
We consider LTL-constrained PO as a systematic framework, decoupling task specification from policy selection, and as an alternative to the standard of cost shaping.
With access to a generative model, we develop a model-based approach that enjoys a sample complexity analysis for guaranteeing both task satisfaction and cost optimality (through a reduction to a reachability problem).
Empirically, our algorithm can achieve strong performance even in low-sample regimes
Introduction
The standard reinforcement learning (RL) framework aims to find a policy that minimizes a cost function.
The premise is that this scalar cost function can completely capture the task specification (known as the “reward hypothesis” [55, 53]).
To date, almost all theoretical understanding of RL is focused on this cost minimization setting (e.g., [62, 32, 31, 57, 45, 9, 24, 19, 10, 3, 4, 40, 47, 48]).
However, capturing real-world task specifications using scalar costs can be challenging.
For one, realworld tasks often consist of objectives that are required, as well as those that are merely desirable.
By combining these objectives into scalar costs, one erases the distinction between these two categories of tasks.
Also, there is recent theoretical evidence that certain tasks are simply not reducible to scalar costs [1] (see Section 2).
In practice, one circumvents these challenges using heuristics such as adding “breadcrumbs” [54].
However, such heuristics can lead to catastrophic failures in which the learning agent ends up exploiting the cost function in an unanticipated way [49, 61, 28, 68, 44].
In response to these limitations, recent work has studied alternative RL paradigms that use Linear
Temporal Logic (LTL) to specify tasks (see Section 7).
LTL is a modeling language that can express desired characteristics of future paths of the system [11].
The notation is precise enough to allow the specification of both the required and desired behaviors; the cost minimization is left only to discriminate between which LTL-satisfying policy is “best”.
This ensures that the main objective — e.g., time, energy, or effort — does not have any relation to the task and is easily interpretable.
Existing work on RL with LTL constraints tends to make highly restrictive assumptions.
Examples include (i) known mixing time of the optimal policy [23], (ii) the assumption that every policy satisfies the task eventually [64], or (iii) known optimal discount factor [26], all of which assist in task satisfaction verification.
These assumptions have complex interactions with the environment, making them impractical if not impossible to calculate.
The situation is made more complex by recent theoretical results [66, 7] that show that there are LTL tasks that are not PAC-MDP-learnable.
In this paper, we address these limitations through a novel policy optimization framework for RL under LTL constraints.
Our approach relies on two assumptions that are significantly less restrictive than those in prior work and circumvent the negative results on RL-modulo-LTL: the availability of a generative model of the environment and a lower bound on the transition probabilities in the underlying MDP.
Under these assumptions, we derive a learning algorithm based on a reduction to a reachability problem.
The reduction in our method can be instantiated with several planning procedures that handle unknown dynamics [12, 46].
We show that our algorithm offers strong constraint satisfaction guarantees and give a rigorous sample complexity analysis of the algorithm.
In summary, the contributions of this paper are:
We provide a novel approach to LTL-constrained RL that requires significantly fewer assumptions, and offers stronger guarantees, than previous work.
We develop several new theoretical tools for our analysis.
These may be of independent interest.
We empirically validate using both infinite- and indefinite-horizon problems, and with composite specifications such as collecting items while avoiding enemies.
We find that our method enjoys strong performance, often requiring many fewer samples than our worst-case guarantees.
Motivating Examples
We examine two examples where standard cost engineering cannot capture the task (Figure 1).
We consider the undiscounted setting here.
See [41, 1] for difficult examples for the discounted setting.
Example 1 (Infinite Loop).
A robot is given the task of perpetually walking between the coffee room and the office (Figure 1 (Left)).
To achieve this behavior, both the policy and cost-function must be history-dependent.
These can be made Markovian through proper state-space augmentation and has been studied in hierarchical reinforcement learning or learning with options [38, 56].
Options engineering is laborious and requires expertise.
Nevertheless, without the appropriate augmentation, any cost-optimal policy of a Markovian cost function will fail at the task.
We will see in Sectionthat any LTL expression comes with automatic state-space augmentation, requiring no expert input.
Example 2 (Safe Delivery).
The goal is to maximize the probability of safely sending a packet from one computer to another (Figure 1 (Right)).
Policy 1 leads to a hacker sniffing packets but passing them through, and is unsafe.
Policy 2 leads to a hacker stealing packets with probability p > 0 , and is safe with probability 1 − p , and is the policy that satisfies the task.
For cost engineering, let R and S be the recurring costs of the received and stolen states.
For the two policies, the avg.
costs are g 1 =
R and g 2 = pS + (1 − p ) R .
Strangely, we must set R > S in order for g 2 < g 1 .
Fortunately, optimizing any cost function constrained to satisfying the LTL specification does not suffer from this counter intuitive behavior as only policy 2 has any chance of satisfying the LTL expression. p 1-p R S
Without proper state-space augmentation, a function cannot Delivery specification is to being interfered Policy should be One would need to penalize receiving the packet significantly over having it stolen:
>  Background and Problem Formulation We now formulate the problem.
An atomic proposition is a variable that takes on a truth value.
An alphabet over a set of atomic propositions AP is given by Σ = 2 AP .
For example, if AP = { a, b } then Σ = {{} , { a } , { b } , { a, b }} .
∆( X ) represents the set of probability distributions over a set X .
3.1 MDPs with Labelled State Spaces
We assume that the environment follows the finite Markov Decision Process (MDP) framework given by the tuple M = ( S M , A M , P M , C M , d M 0 , L M ) consisting of a finite state space S M , a finite action space A M , an unknown transition function P M : S M × A M
→ ∆( S M ) , a cost function
C : S M ×A M → ∆([ c min , c max ]) , an initial state distribution d 0 ∈ ∆( S M ) , and a labelling function L M :
S M → Σ .
We take A M ( s ) to be the set of available actions in state s .
Unlike traditional
MDPs, M has a labeling function L M which returns the atomic propositions that are true in that state.
A run in M is a sequence of states τ =
( s 0 , s 1 , . . . )
reached through successive transitions.
3.2 Linear Temporal Logic (LTL), Synchronization with MDPs, and Satisfaction
Now we give some basic background on LTL.
For a more comprehensive overview, see [11].
Definition 3.1 (LTL Specification, φ ) .
An LTL specification φ is the entire description of the task, including both desired and required behaviors, and is constructed from a composition of atomic propositions, including logical connectives: not ( ¬ ), and ( & ), and implies ( → ); and temporal operators: next ( X ) , repeatedly/always/globally ( G ), eventually ( F ), and until ( U ).
Examples.
Consider again the examples in Section 2.
For AP = { a, b } , some basic task specifications include safety ( G ¬ a ), reachability ( Fa ), stability ( FGa ), response ( a → Fb ), and progress ( a & XFb ) .
For the Infinite Loop example (Figure 1 (Left)), AP = { o, c } indicating the label of the grid location of our agent (office, coffee, or neither).
The specification is “ GF ( o & XFc ) ” meaning “go between office and coffee forever”, and is a combination of safety, reachability, and progress.
For the Safe Delivery example (Figure 1 (Right)), AP = { s } indicating the safety of a state.
The specification is “ Gs ” meaning “always be safe”.
LTL Satisfaction: Synchronizing MDP with LTL.
By synchronizing an MDP with an LTL formula, we can easily check if a run in the MDP satisfies a specification φ .
In particular, it is possible to model the progression of satisfying φ through a specialized automaton, an LDBA B φ [52], defined below.
More details for constructing LDBAs are in [25, 11, 35].
We drop φ from B φ for brevity.
Definition 3.2.
(Limit Deterministic Büchi Automaton, LDBA [52])
An LDBA is a tuple B = ( S B , Σ ∪ A B , P B , S B∗ , s B 0 ) consisting of (i) a finite set of states S B , (ii) a finite alphabet Σ = 2 AP ,
A B is a set of indexed jump transitions (iii) a transition function P B : S B × (Σ ∪ A B ) → 2 S B , (iv) accepting states
S B∗ ⊆ S
B , and (v) initial state s B 0 .
There exists a mutually exclusive partitioning of S B = S B D ∪ S B N such that S B∗ ⊆
S B D , and for s ∈ S B D , a ∈ Σ then P B ( s, a )
⊆
S B D and | P B ( s, a ) | = 1 , deterministic.
A B ( s ) is only (possibly) non-empty for s ∈ S B D and allows B to transition without reading an AP.
A path σ = ( s 0 , s 1 , . . . )
is a sequence of states in B reached through successive transitions. B accepts a path σ
if there exists some state s ∈ S B∗ in the path that is visited infinitely often.
We can now construct a synchronized product MDP from the interaction of M and B .
Definition 3.3.
(Product MDP)
The product MDP X M , B = ( S , A , P, C , d 0 , L, S ∗ ) is an MDP with S = S M × S B , A =
A M ∪ A B , C
(( m, b ) , a ) =
C M ( m, a ) if
a ∈ A M ( m ) otherwise 0 , d 0 = { ( m, b ) | m ∈
d M 0 , b ∈ P B ( s B 0 , L M ( m )) } , L (( m, b ))
=
L M ( m ) , S ∗ = { ( · , b ) ∈ S| b ∈ S B∗ } accepting states, and P : [Equation] A [Equation]
∆ [Equation] S ) taking the form:
[Equation]
A run τ = ( s 0 , s 1 , . . . )
=
(( m 0 , b 0 ) , ( m 1 , b 1 ) , . . . )
in X is accepting (accepted) if ( b 0 , b 1 , . . . ) , the projection onto B , is accepted.
Equivalently, some s ∈ S ∗ in X is visited infinitely often.
This leads us to the following definition of LTL satisfaction:
Definition 3.4 (Satisfaction, τ | = φ ) .
A run τ in X satisfies φ , denoted τ | =
φ , if it is accepted.
Definition 3.5.
(Satisfaction, π | = φ )
A policy π ∈ Π satisfies φ with probability P [ π | = φ ] = E τ ∼ T Pπ [ 1 τ | = φ ] .
Here, 1 X is an indicator variable which is 1 when X is true, otherwise 0 .
T Pπ is the set of trajectories induced by π in X with transition function P .
3.3 Problem Formulation
Our goal is to find a policy that simultaneously satisfies a given LTL specification φ with highest probability (probability-optimal) and is also optimal w.r.t.
the cost function of the MDP.
We consider
(stochastic) Markovian policies Π , and define the set of all probability-optimal policies as Π max = { arg max π ′ ∈
Π P [ π ′ | = φ ] } .
We first define the gain g (average-cost) and transient cost J :
[Equation]
where κ τ is the first (hitting) time the trajectory τ leaves the transient states induced by π .
When P is clear from context, we abbreviate g Pπ and J Pπ by g π and J π , respectively.
Gain optimality for infinite horizon problems has a long history in RL [12, 46].
Complementary to gain optimality, we consider a hybrid objective including the transient cost.
For any λ ≥ 0 , the optimal policy is the probability-optimal policy with minimum combined cost:
[Equation]
In other words, probability-optimal policies are those that satisfy the entirety of the task, both desired and required behaviors, whereas V Pπ,λ ≡ ( J π + λg π ) P [ π | = φ ] is the normalized value function 1 , corresponding to a notion of energy or effort required, with λ representing the tradeoff between gain and transient cost.
We will often omit the dependence of V on P and λ for brevity.
Example.
Consider the Safe Delivery example (Figure 1 (Right)).
For policy 1 , P [1 | = φ ] =
and so 1 ̸∈ Π max .
Let policy 2 be a cost 1 timestep before stolen or receipt, then g 2 =
R is the (conditional) gain, J 2 = 1 is the (conditional) transient costs, P [2 | = φ ] = 1 − p , and V 2 =
(1 + λR )(1 − p ) .
Problem 1 (Planning with Generative Model/Simulator) .
Suppose access to a generative model of the true dynamics P from which we can sample transitions s ′ ∼ P ( s, a ) for any state-action pair ( s, a ) ∈
S × A . 2 With probability 1 − δ , for some errors ϵ φ , ϵ V > 0 , find a policy π ∈ Π that simultaneously has the following properties: ( i )
[Equation]
Approach 4.1 End Components & Accepting Maximal End Components
Our analysis relies on the idea of an end component: a recurrent, inescapable set of states when restricted to a certain action set.
It is a sub-MDP of a larger MDP that is probabilistically closed.
Definition 4.1.
(End Component, EC/MEC/AMEC [11]) Consider MDP ( S , A , P, C , d 0 , L, S ∗ ) .
An end component ( E, A E ) is a set of states E ⊆ S and acceptable actions A E ( s ) ⊆
A ( s ) (where s ∈ E ) such that ∀ ( s, a ) ∈
E ×
A E then Post ( s, a ) = { s ′ | P ( s, a, s ′ ) > 0 } ⊆ E .
Furthermore, ( E, A E ) is strongly connected: any two states in E is reachable from one another by means of actions in A E .
We say an end component ( E, A E ) is maximal (MEC) if it is not contained within a larger end component ( E ′ , A E ′ ) ,
ie. ∄ ( E ′ , A E ′ )
EC where E ⊆ E ′ , A E ( s ) ⊆
A E ′ ( s ) for each s ∈ A .
A MEC ( E, A E ) is an accepting MEC (AMEC) if it contains an accepting state, ∃ s ∈ E s.t. s ∈ S ∗ .
4.2 High-Level Intuition
The description of our approach, LTL Constrained Planning (LCP), in Section 4.4 is rather technical in order to yield theoretical guarantees.
We thus first summarize the high-level intuitions.
Solution Decomposition.
Consider the accepting states s ∗ 1 , s ∗ 2 in Figure 2 (Left), which are the states we need to visit infinitely often to satisfy the specification.
First, let us identify the accepting maximal end components (AMECs) of s ∗ 1 and s ∗ 2 : the state sets A 1 and A 2 (resp.) and their corresponding action sets A A 1 and A A 2 (the blue arrows in A 1 and A 2 ).
Note that these AMECs do not include the yellow action in Figure 2 (Left), which has a chance of leaving A 1 and getting stuck in A 3 .
Our solution first runs a transient policy until reaching A 1 or A 2 , and then switches to a (probabilityoptimal) recurrent policy that stays within A 1 or A 2 (resp.)
while visiting s ∗ 1 or s ∗ 2 (resp.) infinitely often.
A probability-optimal recurrent policy will select actions in A A 1 and A A 2 to visit s ∗ 1 , s ∗ [Figure] infinitely often (e.g., the uniform policies with the AMECs ( A 1 , A A 1 ) and ( A 2 , A A 2 ) ).
Finding a transient policy from s 0 to A 1 , A 2 can be viewed as a reachability problem, which we can solve via a
Stochastic Shortest Path (SSP) problem and leverage recent literature [58, 34].
Cost Optimality.
As stated in OPT, the goal is to find a cost-optimal policy within the set of probability-optimal policies.
For instance, the uniform policy over A A 1 and A A 2 (the blue arrows in Figure 2 (Left) is probability optimal, but may not be cost optimal.
Similarly, the unconstrained cost-optimal policy may not be probability optimal.
Consider just A 1 for the moment.
Suppose the cost of the arrows between the white nodes is 4 while the other costs are 7 .
Then the uniform (probability-optimal) policy in A 1 over A A 1 has cost 12 (cid:0) 4+42 (cid:1) + 12 (cid:0)
+7+ (cid:1)
= 5 .
The gain-optimal policy that deterministically selects the actions between the white nodes ˜ π has cost (cid:0) 4+42 (cid:1) = 4 , but is not probability optimal.
If we perturb ˜ π to make it even slightly stochastic (but still mostly deterministic, i.e η -greedy with η ≈ 0 ), then it will be arbitrarily close to gain optimality and also recover probability optimality.
This is a preferable probability-optimal policy over the uniform policy.
Overall Procedure.
The high-level procedure is: (i) identify the AMECs (e.g. ( A 1 , A A 1 ) , ( A 2 , A 2 )) by filtering out bad actions like the yellow arrow; (ii) find a cost-optimal (optimal gain cost) recurrent policy in each AMEC that visits some s ∗ infinitely often; (iii) instantiate an SSP problem that finds a cost-optimal (optimal transient cost) transient policy from s 0 to A 1 ∪ A 2 and avoids A 3 ; (iv) return a policy that stitches together the policies from ( ii ) and ( iii ) .
See Section 4.4 for the algorithmic details.
We show in Section 5 that this solution gives the optimal solution to OPT.
4.3 Additional Assumptions and Definitions
Perhaps surprisingly, when planning with a simulator (i.e., generative model), even infinite data is insufficient to verify an LTL formula without having a known lower-bound on the lowest nonzero probability of the transition function P [41].
Without this assumption, LTL constrained policy learning is not learnable [66].
We thus begin by assuming a known lower bound on entries in P .
Assumption 1 (Lower Bound) .
We assume we have access to a lower bound β > 0 on the lowest non-zero probability of the transition function P (Sec. 3.1):
[Equation]
We assume that all the costs are strictly positive, avoiding zero-cost (or negative-cost) cycles that trap a policy.
Leveraging cost-perturbations and prior work [58] can remove the assumption.
Assumption 2 (Bounds on cost function) .
The minimum cost c min > 0 (Sec. 3.1) is strictly positive.
Let D =
( s, a, s ′ ) } be all the collected samples ( s, a, s ′ ) while running the algorithm.
At any point, (cid:98) P ( s, a, s ′ ) =
|{ ( s,a,s ′ ) ∈ D }| |{ ( s,a ) ∈ D }| is the empirical frequency of visiting s ′ from ( s, a ) .
We introduce an event E and error ψ ( n ) to quantify uncertainty on (cid:98) P ( s, a, s ′ )
based on current data: n ( s, a ) = |{ ( s, a ) ∈ D }| .
E is based on empirical Bernstein bounds [42], and holds w.p. 1 − δ
(Lemma B.1).
Definition 4.2 (High Probability Event) .
A high probability event E :
[Equation]
where ψ
[Equation]
( n ) ≡ [Equation] ξ ( n ) ≡ log( 4 n 2 |S| 2 |A| δ ) / ( n − 1) .
Remark 4.1.
For some ρ > 0 , if we require | P ( s, a, s ′ ) − (cid:98) P ( s, a, s ′ )
| ≤ ρ then we need n ( s, a ) =
ψ − 1 ( ρ ) samples for state-action pair ( s, a ) .
See Lemma B.2 for the quantity ψ − 1 ( ρ ) .
Definition 4.3 (Plausible Transition Function) .
The set of plausible transition functions is given by
P = { ˜ P : S ×
A →
[Equation]
S ) | [Equation] Let P ( s, a ) ≡ { P ( s, a, · ) | P ∈ P} be the possible transition distributions for state-action pair ( s, a ) .
We denote P
π ( s, s ′ ) =
E a ∼ π [ P ( s, a, s ′ )] as the Markov chain given dynamics P with policy π , and can be thought of as a |S| × |S| matrix P π = { p ij } |S| i,j =1 .
4.4 Main Algorithm: LTL Constrained Planning (LCP)
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
[Table]
[Table] [Table]
[Table]
Our approach, LTL Constrained Planning (LCP), has three components, as shown in Algorithm 1 and described below.
Recall from Problem 1 that the policy optimization problem OPT is instantiated over a product
MDP (Def. 3.3), and that we are given a generative model of the true dynamics P from which we can sample transitions s ′ ∼ P ( s, a ) for any state/action pair.
Finding AMECs ( FindAMEC ).
After sampling each state-action pair ϕ FindAMEC
=
O ( 1 β ) times (see
Prop. B.4), by Assumption 1, we can verify the support of P .
We can compute all of the MECs using
Algorithm 47 from [11].
Among these MECs, we keep the AMECs, which amounts to checking if the MEC ( A i , A A i ) contains an accepting state s ∗ ∈ S ∗ from the given product MDP.
PlanRecurrent ( PR ).
To plan in each AMEC ( A, A A ) (i.e., find the optimal recurrent Algorithm 2 PlanRecurrent ( PR ) Param: AMEC ( A, A A ) , error ϵ PR > [Table]
ρ
[Table] [Table] FindAMEC ( β )) //
ρ
∼
{+∥ P − ˜ P ∥ − 
+}
[Table]
: repeat
[Table]
Set
[Table]
ρ
[Table]
−
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
return
policy)
, we use Alg. 2 with (extended) relative value iteration ( VI , Alg. 4 in appendix) using the optimistic Bellman operator L α PR (see Table 1, we discuss α in next paragraph).
Let π v denote the greedy policy w.r.t.
the fixed point v =
L α PR v ( v is the optimistic value estimate).
Using the η -greedy policy, π ≡ (1 − η ) π v + η Unif ( A A ) (Alg. 2, Line 7), together with P π , makes A recurrent: s ∗ ∈
A is visited infinitely often and P [ π | = φ | s 0 ∈ A ] = 1 .
Since η can be arbitrarily small (Lemma B.7), then g π ≈ g π v and π is both cost and probability optimal.
As intuited in Section 4.2, π has full support over A A but is nearly deterministic.
[Table]
VI in Line 5 of Alg.
2 is an iterative procedure (Alg. 4 in appendix), and terminates via d PR < ϵ L PR (Table 1).
Convergence of extended VI is guaranteed [46, 29, 22], so long as the dynamics, ˜ P = arg min p ∈P ( s,a ) p T v , achieving the inner minimization of L α PR are aperiodic – hence the aperiodicity transform α ∈ (0 , 1) in L α PR [46].
Computing ˜ P can be done efficiently [29] (Alg. 5 in appendix).
For stability, we shift each entry of v n by the value of the first entry v n (0)
Alg.
2 returns the average gain cost g π of policy π when we have enough samples for each state-action pair in ( A, A A ) to verify that n > ψ
[Table] (cid:17) where ∆( ˜ P π ) = 12 max ij (cid:80) k | ˜ p
ik − ˜ p jk | .
Here, ∆( ˜ P π ) is an easily computable measure on the ergodicity of the Markov chain ˜
P π [18].
We track ψ ( n ) (recall Def. 4.2) via a variable ρ and sample ψ − 1 ( ρ ) ≈ 1 ρ 2 (see Lemma B.2) samples from each state-action pair in ( A, A A ) (Alg. 2, Line 4).
We halve ρ each iteration (Alg. 2, Line 3) and convergence is guaranteed because ρ will never fall below some unknown constant ϵ PR (1 − ¯∆ A ) {+
+} | A | c max (see Lemma B.8); the halving trick is required because ¯∆ A is unknown a priori.
Proposition 4.2 ( PR Convergence & Correctness, Informal) .
Let π A be the gain-optimal policy in
AMEC ( A, A ) .
Algorithm 2 terminates after at most log 2 (cid:16) 6 | A | c max ϵ PR (1 − ¯∆ A ) (cid:17) repeats, and collects at most n = ˜
O ( | A | 2 c 2max ϵ 2 PR (1 − ¯∆ A ) 2 ) samples for each ( s, a ) ∈
( A, A A ) .
The η -greedy policy π w.r.t.
v ′
(Alg. 2, Line 5) is gain optimal and probability optimal: | g π − g π A | < ϵ PR , P [ π | = φ | s 0 ∈ A ] = 1 . Algorithm
[Table]
[Table]
PlanTransient ( PT ).
This is the stochastic shortest path (SSP) reduction step that finds a policy from the initial state s 0 to the
AMECs (Alg. 3).
The main algorithmic tool used by PlanTransient is similar to that of
PlanRecurrent : it also uses extended value iteration ( VI , Alg. 4 in appendix) but with a different optimistic Bellman operator L PT (Table 1), and then returns a (fully deterministic) greedy policy w.r.t.
the resulting optimistic value v (Alg. 3, Line 4).
L PT is used to calculate the highest probability, lowest cost path to the AMECs (Alg. 3, Line 3).
Since rejecting end components might exist (see A 3 from Figure 2 (Left)), a trajectory may end up stuck and accumulate cost indefinitely, and so we must bound ∥ v ∥ ∞ <
¯ V /ϵ φ to prevent blow up.
In Prop.
B.13, we show how to select ¯ V such that π will reach the target states (in this case, the
AMECs), first with high prob and then with lowest cost.
The existence of such a bound on ∥ v ∥ ∞ was shown to exist, without construction, in [34].
In practice, choosing a large ¯ V is enough.
The terminal costs V T (Alg. 3, Line 1) together with Bellman equation L PT has value function ˜ V π ≈ p
( J π + 1 p (cid:80) ki =1 p i
g π i )
+ (1 − p ) ¯ V /ϵ
φ ≈ V π , relating to V π (OPT), see Section A.1.
Here, p i = P [ π reaches A i ] ≡ E τ ∼ T Pπ [1 ∃ s ∈
τ s.t s ∈
A i ] and p = (cid:80) ki
=1 p i .
VI converges when d PT <
ϵ PT (see Table 1).
Convergence of extended VI for SSP is guaranteed [58, 34].
The number of samples required for each state-action pair ( s, a ) ∈ ( S \ ∪ A i ) × A is ϕ PT = ψ − 1 (cid:16)
[Table]
(cid:17) .
Proposition 4.3 ( PlanTransient Convergence & Correctness, Informal) .
Denote the cost- and prob-optimal policy as π ′ .
After collecting at most n = ˜ O ( |S\∪ ki =1 A i
[Equation]
) samples for each ( s, a ) ∈ ( S \ ∪ ki =1 A i ) × A , the greedy policy π w.r.t.
v ′
(Alg. 3, Line 3) is both cost and probability optimal:
[Equation]
End-To-End Guarantees
The number of samples necessary to guarantee an ( ϵ V , ϵ φ , δ ) -PAC approximation to the cost-optimal and probability-optimal policy relies factors: β (lower bound on the min.
non-zero transition probability of P ), { c min , c max } (bounds on the cost function C ), ¯∆ A i (worst-case coefficient of ergodicity for EC ( A i , A A i ) ), ¯ V (upper bound on the value function), and λ (tradeoff factor).
Theorem 5.1 (Sample Complexity) .
Under the event E , Assumption 1 and 2, after
[Equation]
samples 5 are collected from each state-action pair, the policy π returned by Algorithm 1 is, with probability 1 − δ , simultaneously ϵ V -cost optimal and ϵ φ -probability optimal, satisfying:
[Equation]
With a sufficiently large λ (which may not be verifiable in practice), π is also gain optimal.
Corollary 5.2 (Gain (Average Cost) Optimality) .
There exists λ ∗ > 0 s.t. for λ > λ ∗ , the policy π returned by Alg.
1 satisfies (4) ,
g π = arg min π ′ ∈ Π max g π ′ , and is probability and gain optimal.
The high-level structure of our analysis follows the algorithm structure in Section 4.4, via composing the constituent guarantees.
To complete the analysis, we develop some technical tools which may be of independent interest, including a gain simulation Lemma B.8 and an η -greedy optimality Lemma
B.7.
For ease of exposition, we also ignore paths between AMECs (see Appendix D.2).
Empirical Analysis
We perform experiments in two domains: (1) Pacman domain where an agent find food and indefinitely avoids a ghost; (2) discretized version of mountain car (MC) [14] where the agent must reach the flag.
Our goal is to understand whether: (i) our LCP approach (Alg.1) produces competitive polices; (ii)
LCP can work in continuous state spaces through discretization; (iii) LCP can enjoy efficient sample complexity in practice.
For a baseline, we use Logically Constrained RL (LCRL, [26]), which is a Q-learning approach to LTL-constrained PO in unknown MDPs.
We also do heavy cost shaping to LCRL as another baseline.
See App E for more details, experiments, and figures.
6.1 Results
Competitiveness of the policy in full LTL specs?
The probability of LCP satisfying the LTL [Figure] spec in Figure 3 (Left) approaches 1 much faster than the two baselines.
The returned policy collects the food quickly and then stays close, but avoids, the ghost.
Any policy that avoids the ghost is equally good, as we have not incentivized it to stay far away.
LCRL redefines cost as 1 if the LTL is solved and 0 otherwise, which is too sparse and learning suffers.
Indeed, shaped LCRL performs better than straight LCRL.
Performance in continuous state space?
Similarly, the probability of satisfying the
LTL spec in Figure 3 (Right) goes up to 1 .
However, here the LCRL (shaped) baseline performs relatively well as it is being given “breadcrumbs” for how to solve the task.
Our algorithm performs well without needing any cost shaping.
Standard LCRL fails to learn.
This experiment demonstrates that our method can be used even in discretized continuous settings.
Sample Complexity?
Our theory is quite conservative w.r.t.
empirical performance.
In Pacman (Figure 3, Left), Thm. 5.1 suggests ≈ 350 samples per ( s, a ) pair just to calculate the AMECs.
Empirically, LCP finds a good policy after 11 samples per ( s, a ) pair ( ∼ 66 k/ 6 k samples/pair).
Other Considerations.
One of the strengths and potential drawbacks of LTL is its specificity.
If a φ , for a truly infinite horizon problem, is to “eventually” do something, then accomplishing the task quickly is not required.
As a finite horizon problem, in MC (Fig. 3, Right) SSP finds the fastest path to the goal.
In contrast, since any stochastic policy with full support will “eventually” work, the policy returned by LCP for Fig 1 (Left) (Fig. 2 Center, & App Fig. 7) may take exponential time to complete a single loop.
Two straightforward ways to address this issue are: (a) including explicit time constraints in φ ; and (b) cost shaping to prefer policies reaching some s ∗ quickly and repeatedly.
Unlike standard cost-shaping, φ satisfaction is still guaranteed since the cost is decoupled from φ .
Related Work
Constrained Policy Optimization.
One attempt at simplifying cost functions is to split the desired behaviors from the required behaviors.
The desired behaviors remain as part of the cost function while the required behaviors are treated as constraints.
Recent interest in constrained policy optimization within the RL community has been related to the constrained Markov Decision Process (CMDP) framework [6, 39, 2, 43].
This framework enables clean methods and guarantees, but enforces expected constraint violations rather than absolute constraint violations.
Setting and interpreting constraint thresholds can be very challenging, and inappropriate in safety-critical problems [38].
LTL + RL.
Recently, LTL-constrained policy optimization has been developed as an alternative to
CMDPs [41].
Unlike CMPDs, the entire task is encoded into an LTL expression and is treated as the constraint.
Q-learning variants when dynamics are unknown and Linear Programming methods when dynamics are known are common solution concepts [50, 26, 13, 16, 20].
The Q-learning approaches rely on proper, unknowable tuning of discount factor for their guarantees.
Theoretically oriented works include [23, 64].
While providing PAC-style guarantees, the assumptions made in these works rely on unknowable policy-environment interaction properties.
We make no such assumptions here.
Another solution technique is employing reward machines [60, 17, 63] or high-level specifications that can be translated into reward machines [30].
These works are generally empirical and handle finite or repeated finite problems (episodic problems at test time); they can only handle a smaller set of LTL expressions, specifically regular expressions.
Our work handles ω -regular expressions, subsuming regular expressions and requires a nontrivial leap, algorithmically and theoretically, to access the broader set of allowable expressions.
Many problems are ω -regular problems, but not regular, such as liveness (something good will happen eventually) and safety (nothing bad will happen forever).
The works that attempt to handle full LTL expressibility redefine reward as 1 if the LTL is solved and 0 otherwise; the cost function of the MDP is entirely ignored.
Verification and Planning.
As an alternative to our approach, one might consider LTL satisfaction verification and extend it to an optimization technique by checking every policy (which will naively take an exponential amount of samples to verify a single policy [15, 8]).
Many verification approaches exist [36, 11, 5, 67, 37, 27] and among the ones that do not assume known dynamics, the verification guarantees rely on quantities as difficult to calculate as the original verification problem itself [8].
Discussion
We have presented a novel algorithm, LCP, for policy optimization under LTL constraints in an unknown environment.
We formally guarantee that the policy returned by LCP simultaneously has minimal cost with respect to the MDP cost function and maximal probability of LTL satisfaction.
Our experiments verify that our policies are competitive and our sample estimates conservative.
The assumptions we make are strong, but to the best of our knowledge, are the most relaxed amongst tractable model-based algorithms proposed for this space.
Model-free algorithms (Q-learning) have less stringent assumptions but do not come with the kind of guarantees that our work has and largely ignore the cost function, solving only part of the problem.
An interesting future direction would be to extend our work to continuous state and action spaces and settings with function approximations.
Acknowledgements.
Cameron Voloshin is funded partly by an NSF Graduate Fellowship and a Kortschak
Fellowship.
This work is also supported in part by NSF #1918865, ONR #N00014-20-1-2115, and NSF #2033851.