Finding Safe Zones of Markov Decision Processes Policies
Abstract
Safety is essential for gaining trust in Markov Decision Process’s policies.
We suggest a new method to improve safety, using S AFE Z ONES .
Given a policy, we define its S AFE Z ONE as a subset of states, such that most of the policy’s trajectories are confined to this subset.
A trajectory not entirely inside the S AFE Z ONE is potentially unsafe and should be examined.
The quality of the S AFE Z ONE is parameterized by the number of states and the escape probability, i.e., the probability that a random trajectory will leave the subset.
S AFE Z ONES are especially interesting when they have a small number of states and low escape probability.
We study the complexity of finding optimal S AFE Z ONES , and show that in general, the problem is computationally hard.
For this reason, we concentrate on computing approximate S AFE Z ONES .
Our main result is a bi-criteria approximation algorithm which gives a factor of almost 2 approximation for both the escape probability and
S AFE Z ONE size, using a polynomial size sample complexity.
Introduction
Most research in reinforcement learning (RL) deals with the problem of learning an optimal policy for some Markov decision process (MDP).
One notable exception for that is Safe RL, that focuses on finding the best policy that meets safety requirements.
Typically, these problems are handled by adjusting the objective to include safety requirements and then optimizing over it or incorporating additional safety constraints to the exploration stage.
Anomaly Detection is the problem of identifying patterns in data that do not correspond to what is expected, i e., anomalies.
Anomaly Detection addresses a variety of applications: cyber-security, fraud detection, failure detection, etc.
(see
Chandola et al.
(2009) for survey).
In this paper we introduce the S AFE Z ONE problem, a general approach for safe RL and anomaly detection that concentrates on a given policy rather than finding a policy that follows some predefined safety specifications and emphasizes entire trajectories in order to detect anomalies.
Consider a policy for a finite horizon MDP.
The policy induces a Markov Chain (MC) on the MDP.
Given a subset of states, we define the escape probability to be the probability that a random trajectory has at least one state outside this subset (hence the trajectory escapes it).
A S AFE Z ONE is a subset of states whose quality is measured by its’ size and escape probability (ideally, both are small).
If a
S AFE Z ONE has low escape probability, we consider it safe .
Trivial S AFE Z ONE solutions are the entire set of states (which has minimal escape probability of 0 on the account of maximal size), and the empty set (which has minimal size but has maximal escape probability of 1 ).
We are interested to find S AFE Z ONE with a good tradeoff: namely a relatively small set size with small escape probability.
More precisely, given a bound over the escape probability, ρ > 0 , the goal of the learner is, using trajectory sampling, to find the smallest
S AFE Z ONE with escape probability at most ρ .
We address unknown environment, by which we mean no prior knowledge on the transition function or the policy used.
The learner can only access random trajectories generated by the induced MC.
For many applications, if there exists such a small
S AFE Z ONE it is useful to find it.
Consider for example automatic robotic arm that assembles products.
If something unusual happened during the assembly of a product, it might result in a malfunctioning product.
In that case, the operator should be notified (anomaly detection).
On the other hand, we would not like to call the operator too often.
If we find a S AFE Z ONE , we can make sure that we notify the operator only in the rare events the production process (trajectory) escapes it.
Furthermore, if the S AFE Z ONE is small, the manufacturer can potentially test the S AFE Z ONE states and verify their compliance, ensuring that the majority of products are well constructed for a significantly lower testing budget.
Another useful application is transportation design.
For example, given data regarding bicycle commutes (not necessarily done on bicycle lanes) in a populated areas, pave bike lanes in the
S AFE Z ONE , namely in a way that would accommodate popular commutes, from starting point to destination.
Making cycling safer and more accessible would also promote it as a viable transportation option, which in turn benefits the environment Watkins et al.
We remark that finding a S AFE Z ONE alone does not suffice for safety;
Rather, a nearly optimal
S AFE Z ONE is a behavioral description that can be used for safety applications, such as safer cycling.
As another example, efficient testing (of states within the S AFE Z ONE ) that “captures” most of the products’ assembly process would improve safety.
Other motivations include imitation learning with compact policy representation.
Namely, design a smaller state policy that preforms well for most cases but might be undefined on some states.
In this case, trajectories that reach undefined states have zero reward, and such trajectories are captured by the escape probability.
One natural application for that is creating a ‘lite’ version for a given software such as Microsoft’s Windows Lite.
Our work can also be viewed through the lens of explainable RL, where the goal is to explain a specific policy.
S AFE Z ONE is a new post-hoc explanation of the summarization type (Alharin et al., 2020).
Going back to the bicycle example, a municipality could provide a convincing explanation to its community for the chosen design.
Our results are the following:
Introducing the S AFE Z ONE problem (Section 2), and some of its applications.
We explore naive approaches, namely greedy algorithms that select S AFE Z ONES based on state distributions and trajectory sampling.
In addition, we show cases in which their solutions are far from optimal, either in terms of high escape probability or significantly larger set size (see Section 3).
We design F INDING S AFE Z ONE , an efficient approximation algorithm with provable guarantees.
The algorithm returns a S AFE Z ONE which is slightly more than twice in terms of both the size and the escape probability compared to the optimal (see Section 4).
For brevity, some algorithms and (full) proofs are relegated to the appendix.
Trajectory escaping.
The S AFE Z ONE problem deals escaping trajectories.
In particular, given a
S AFE Z ONE , a trajectory escapes it, no matter if only one of its states is outside the S AFE Z ONE or all of them.
A related, yet very different problem, is that of minimizing a subset size, such that the expected number of states outside the set is minimized.
This related problem, while significantly easier (as it is solved by returning the most visited states), does not apply to the applications we described earlier.
In Section 3, we show that the solution for the S AFE Z ONE does not necessarily overlaps with the most visited states.
Furthermore, simply returning states which appeared in trajectory samples could result in a set size far from optimal.
1.1 Related Work
MDPs have been studied extensively in the context of decision making in particular by the Reinforcement Learning (RL) community (see Puterman (1994) for a broad background on MDPs, and Sutton & Barto (2018) for background on reinforcement learning).
Safe RL.
A related line of research is safe RL, where the learner’s goal is to find the best policy that satisfies safety guarantees.
The two main methodologies to handle such problems are: (1) altering the objective to include the safety requirement and optimizing over it, and (2) adding safety constraints to the exploration part.
See Pfrommer et al.
; Emam et al.
; Xu et al.
; Hendrycks et al. ; HasanzadeZonuzy et al.
(2021) for recent works and García & Fernández (2015);
Amodei et al.
(2016) for surveys.
In our work, the goal is not to find the optimal policy, but instead given a policy, finding its S AFE Z ONE .
Moreover, the S AFE Z ONE problem is not characterized by specific requirements, and beyond the MDP, the solution could very much depend on the given policy.
Imitation Learning.
In imitation learning, the learner observes a policy behaviour and wants to imitate it (see Hussein et al.
(2017) for survey).
Similar to imitation learning, we are given access to samples of a given policy.
In contrast, rather than imitating the policy we find the policy’s S AFE Z ONE , which is an important property of the policy.
Approximate MDP equivalence.
Another related research line is that of finding an (almost) equivalent minimal model for a given MDP, where the goal is that the optimal policy on the (almost) equivalent model induces an (approximately) optimal policy in the original MDP, e.g., Givan et al. (2003); Even-Dar & Mansour (2003).
This line of works and ours differ in that we do not try to modify the MDP (e.g., cluster similar states), but rather to find a S AFE Z ONE , a property which is defined for the existing MDP and a specific policy.
Explainability.
In explainability, the goal is to provide a post-hoc explanation to a specific given model Molnar (2019), e.g., using decision trees Blanc et al.
; Moshkovitz et al.
(2021), influential examples Koh & Liang (2017), or a local approximation explanations Li et al.
We focus on explainability for reinforcement learning, and specifically we suggest a new summarization explanation through our S AFE Z ONE , Amir & Amir (2018).
MC with traps.
A decision problem that might seem related to ours is that of MC with traps (den
Hollander et al.
Given an input of a MC (with possibly infinite state space), a starting state, and states trapping (absorbing) probabilities, the goal is to decide whether or not a (possibly infinite) random walk would reach an absorbing state with probability 1 , or not.
The Safe Zone Problem
We model the problem using a Markov model with finite horizon H > 1 .
Formally, there is a Markov chain (MC) ⟨S , P, s 0 ⟩ where S is the set of states, s 0 ∈ S is the initial state and P : S × S → [0 , 1] is the transition function that maps a pair of states into probability by P ( s, s ′ )
=
Pr[ s t +1 = s ′
| s t = s ] .
We assume the transition function P is induced by a policy π : S
→ Simplex A on an MDP ⟨S , s 0 , P ′ , A⟩ with transition function P ′ :
S ×
A × S → [0 , 1] such that P ( s, s ′ )
= (cid:80) a ∈A P ′ ( s, a, s ′ )
· π ( a | s ) for all s, s ′ ∈ S (though any MC can be generated this way, thus our theoretical guarantees apply for general MCs).
A trajectory τ = ( s 0 , . . .
, s H ) starts in the initial state s 0 and followed by a sequence of H states generated by P , i.e., Pr[ s i +
= s ′
| s i = s ]
= P ( s, s ′ ) for all i ∈
[ H ] , where [ H ] := { 1 , . . .
, H } .
We abuse the notation and regard a trajectory τ both as a sequence and a set.
Given a subset of states F ⊆ S , a trajectory τ escapes F if it contains at least one state s ∈ τ such that s / ∈ F , i.e., τ ̸⊆ F .
We refer to the probability that a random trajectory escapes F as escape probability and denote it by ∆( F ) =
Pr τ [ τ ̸⊆ F ] .
We call F a ρ − safe (w.r.t.
the model ⟨S , s 0 , P ⟩ ) if its escape probability, ∆( F ) , is at most ρ .
Formally,
Definition 2.1.
A set F ⊆ S is ρ − safe if
[Equation]
where τ is a random trajectory.
A set F ⊆ S is called ( ρ, k ) − S AFE Z ONE if F is ρ − safe and | F | ≤ k .
Given a safety parameter ρ ∈ (0 , 1) , we denote the smallest size ρ − safe set by k ∗ ( ρ ) :
[Equation]
Whenever the discussed parameter ρ is clear from the context we use k ∗ instead of k ∗ ( ρ ) .
We remark that there might be multiple different ( ρ, k ) − S AFE Z ONE sets.
The learner knows the set of states, S , the initial state, s 0 , and the horizon H but has no knowledge regarding the transition function P or the minimal size of the ρ − safe set, k ∗ .
Instead, the learner receives information about the model from sampling trajectories from the distribution induced by π .
A LGORITHM
S AFE
S ET S IZE
S AMPLE C OMPLEXITY
G REEDY BY T HRESHOLD 2 ρ k ∗ H/ρ –
S IMULATION 2 ρ O ( k ∗ H ln k ∗ ) poly ( k ∗ , 1 ρ ) G REEDY AT E ACH S TEP * ρH k ∗ – F INDING S AFE Z
ONE 2 ρ + 2 ϵ (2 + δ ) k ∗ poly ( k ∗ , H, 1 ϵ , 1 δ )
Given ρ > 0 , the ultimate goal of the learner would have been to find a ( ρ, k ∗ ( ρ )) − S AFE Z ONE .
However, finding a ( ρ, k ∗ ( ρ ))
− S AFE Z ONE is NP-hard, even when the transition function P is known.
This is why we loosen the objective to find a bi-criteria approximation ( ρ ′ , k ′ )
− S AFE Z ONE .
(Bi-criteria approximations are widely studied in approximation and online algorithms Vazirani (2001);
Williamson & Shmoys (2011).
In our setting, given ρ the objective is to find a set F which is ( ρ ′ , k ′ )
− S AFE Z ONE with minimal size k ′ ≥ k ∗ and minimal escape probability ρ ′ ≥ ρ .
In addition, we are interested in minimizing the sample complexity.
Notice that the learner can efficiently verify, with high probability, whether a set F is approximately ρ − safe or not.
Summary of Contributions.
We summarize the results of all the algorithms that appear in the paper in Table 1.
The bounds of G REEDY BY T HRESHOLD and G REEDY
A T E ACH S TEP requires the
Markov Chain model as input, and a pre-processing step that takes O ( | S | 2 H ) time.
Additionally, the bounds for first three algorithms (the naive approaches) requires an additional knowledge of k ∗ ( ρ ) .
Beyond the upper bounds, we provide instances that show that the upper bounds are tight up to a constant for each of the first three algorithms (the naive approaches).
The following theorem is an informal statement of our main theorem.
Theorem 2.2.
For every ρ, ϵ, δ > 0 , with probability ≥ 0 .
99 there exists an algorithm that returns a set which is (2 ρ + 2 ϵ, (2 + δ ) k ∗ ) − S AFE Z ONE .
The running time of the algorithm is also bounded by poly ( k ∗ , H, 1 δ , 1 ϵ ) .
Gentle Start
This section explains and analyzes various naive algorithms to the S AFE Z ONE problem.
We show that even if the transition function is known in advance, these naive algorithms result in outputs that are far from optimal.
To describe the algorithms, we define for each state s the probability to appear in a random trajectory and denote it by p ( s ) =
Pr τ [ s ∈ τ ] ∈ [0 , 1] .
Note that (cid:80) s ∈S p ( s ) is a number between 1 and H (e.g., p ( s 0 ) = 1 ), and can be estimated efficiently using dynamic programming if the environment and policy are known and sampling otherwise.
To be precise, some of the algorithms assume the probabilities { p ( s ) } s ∈S are received as input.
Greedy by Threshold Algorithm.
The algorithm gets, in addition to ρ , the distribution p and a parameter β > 0 as input.
It returns a set F that contains all states s with probability at least β , i.e., p ( s ) ≥ β .
We formalize this idea as Algorithm 3 in Appendix A. For β = ρk ∗ , the output of the algorithm is (cid:16) 2 ρ, k ∗ Hρ (cid:17) − S AFE Z ONE .
More generally, we prove the following lemma.
Lemma 3.1.
For any ρ, β ∈ (0 , 1) , the G REEDY BY T HRESHOLD
A LGORITHM returns a set that is ( ρ + k ∗ β, Hβ ) − S AFE Z ONE .
In particular, for β = ρk ∗ , this set is (cid:16) 2 ρ, k ∗ Hρ (cid:17) − S AFE Z ONE .
While it is clear why there are instances for which the safety is tight, Lemma A.1 in Appendix A shows that the set size is tight as well.
Simulation Algorithm.
The algorithm samples O
( ln k ∗ β ) random trajectories and returns a set F with all the states in theses trajectories.
It is formalized in Appendix A as Algorithm 4.
Lemma 3.2.
Fix ρ, β ∈ (0 , 1) .
With probability at least 0 .
99 , S IMULATION
Algorithm returns a set that is (cid:16) ρ + k ∗ β, O ( k ∗ + ρH ln k ∗ β ) (cid:17) − S AFE Z ONE .
In particular, for β = ρk ∗ , this set is (2 ρ, O ( k ∗ H ln k ∗ ))
− S AFE Z ONE .
While this algorithm achieves a low escape probability, only 2 ρ , in Lemma A.2 in the appendix we prove that the size of F is tight up to a constant, i.e., an MDP instance where | F | =
Ω( k ∗ H ln k ∗ ).
So far, the presented algorithms were approximately safe (i.e., low escape probability), but might return large subsets.
Without further assumptions, the following algorithm provides a ( ρH, Hk ∗ ) − S AFE Z ONE .
However, when considering MDPs with a special structure it provides an optimal sized S AFE Z ONE , at the price of large escape probability.
Greedy at Each Step Algorithm.
For the analysis of the next algorithm we assume the MDP is layered , i.e., there are no states that appear in more than a single time step and denote S =
(cid:83)
Hi =1 S i .
I.e., the transitions P ( s, s ′ ) are nonzero only for s ′ ∈ S
i +1 and s ∈ S
i .
The G REEDY AT E ACH
S TEP A LGORITHM takes at each time step i the minimal number of states such that the sum of their probabilities is at least 1 − ρ .
It is formalized in Appendix A as Algorithm 5.
Lemma 3.3.
For any ρ ∈ (0 , 1) , if the MDP is layered, G REEDY AT E ACH S TEP A LGORITHM returns a set that is ( ρH, k ∗ ) − S AFE Z ONE .
In Lemma A.3 we have a lower bound on the escape probability, which asymptotically matches.
Weaknesses of the naive algorithms.
We showed algorithms that identify S AFE Z ONE with either escape probability much greater than ρ or with size much greater than k ∗ .
This holds even when providing extra information (such as the transition function and/or the optimal size of the ρ − safe set, i.e., k ∗ ).
Moreover, we showed tight lower bounds for these algorithms.
Algorithm for Detecting Safe Zones
In this section we suggest a new algorithm that builds upon and improves the added trajectory selection of the S
IMULATION Algorithm.
One reason for why S IMULATION returns a large set is that it treats every sampled trajectory identically, regardless of how many states are being added.
More precisely, fix any ( ρ, k ∗ ) − S AFE Z ONE set, F ∗ , and consider a trajectory τ that escapes it, i.e., τ ̸⊆ F ∗ .
If τ was sampled, its states are added to the constructed set F , which might increase the size of F by up to H states that are not in F ∗ , without significantly improving the safety.
In contrast, when selecting which trajectory to add to F , we would consider the number of states it adds to the current set.
For the sake of readability, we refer to any state which is not in the current set
F as new , and denote by new F ( τ ) the number of new states in τ w.r.t.
F , i.e., new F (
[Equation]
Note that for every F ⊆ S , we have that Pr τ [ new F ( τ ) ̸ = 0] =
∆( F ) .
The new algorithm does not sample each trajectory uniformly at random, but sample from a new distribution, which will be denoted by Q F .
While favoring trajectories with higher probabilities, which we already get by the sampling process, another key idea would guide this new distribution:
To prefer trajectories that gradually increase the size of F .
To implement this idea, we will ensure that the probability of adding a trajectory τ to F should be inversely proportional to new F ( τ ) .
Formally, the support of Q F is the trajectories with new states, i.e., X = { τ | new F ( τ ) ̸ = 0 } .
For every τ ∈ X , Q F ( τ ) ∝
Pr[ τ ] new F ( τ ) , where Pr[ τ ] is the probability of trajectory τ under the Markov
Chain with dynamics P .
Note that the new distribution depends on the current set F , and changes as we modify it.
Intuitively, adding trajectories to F according to Q F instead of adding trajectories sampled directly from the dynamics (as we do in S IMULATION ) would increase the expected ratio between the added safety and the number of new states we add to F , thus improving the set size guarantee of the output set.
We elaborate on this in Appendix B.1.2.
Our main algorithm is F INDING S AFE Z ONE , Algorithm 1.
The algorithm receives, in addition to the safety parameter ρ , parameters ϵ, λ ∈ (0 , 1) , and maintains a set F that is initiated to { s 0 } .
On a high level, to implement the idea of adding trajectories to F according to Q F , we use rejection sampling .
Namely, in each iteration of the while–loop we first sample a trajectory τ and if new F ( τ ) ̸ = 0 , we accept it with probability 1 /new F ( τ ) .
If the trajectory is accepted, it is added to F .
More precisely, if new F ( τ ) ̸ = 0 , we sample a Bernoulli random variable, accept ∼ Br (1 /new F ( τ )) .
If accept = 1 , we add τ to F .
This process of adding trajectories to F generates the desired distribution, Q F .
Whenever a trajectory is added to F , we estimate the escape probability ∆( F ) (w.r.t.
the updated set, F ).
The algorithm stops adding states to F and returns it as output when it becomes “safe enough”.
To be precise, let (cid:98) ∆( F ) denote the result of the escape probability estimation (by sampling trajectories and returning the fraction of trajectories escaping F ).
If (cid:98) ∆( F ) ≤ 2 ρ + ϵ , it means that F is (2 ρ + 2 ϵ ) − safe with probability ≥ 1 − λ j > 1 − λ , in which case the algorithm terminates and returns F as output.
To implement the estimation (cid:98) ∆( F ) , the algorithm calls EstSafety Subroutine.
The subroutine samples N j = Θ
( 1 ϵ 2 ln 2 λ j ) trajectories, and returns the fraction of trajectories that escaped F .
Algorithm
Input: ρ ∈ (0 , 1)
Parameters: ϵ, λ ∈ (0 , 1)
F ← { s 0 } , j ← 1 , (cid:98) ∆( F ) ←while (cid:98) ∆( F ) > 2 ρ + ϵ do τ ← sample a random trajectory
Compute new F ( τ ) if new F ( τ ) ̸ = 0 then sample accept ∼ Br (1 /new F ( τ ))
if accept = 1 then
F ← F ∪ τ λ j ← 3 λ 2( jπ ) 2 , j ← j +(cid:98) ∆( F ) ← EstSafety ( ϵ, λ j , F ) end if end if end while return F
Algorithm
Input: subset F
Parameters: ϵ, λ j ∈ (0 , 1) (cid:98) ∆( F ) ←
T ← sample N j = 12 ϵ 2 ln 2 λ j trajectories for τ ∈ T do if τ ̸⊆ F then (cid:98) ∆( F ) ← (cid:98) ∆( F ) + 1 N j end if end for return (cid:98) ∆( F )
Discussion and Open Problems
In this paper, we have introduced the S AFE Z ONE problem.
We designed a nearly (2 ρ, 2 k ∗ ) approximation algorithm for the case where the model and policy are unknown to the algorithm.
Beyond improving the approximation factors (or showing that it cannot be done unless P = NP ), a natural direction for future work is the following.
Given ρ > 0 and an MDP (known or unknown to the learner), find a policy with a small ρ − safe set, with nearly optimal value.
In fact, an efficient solution for this could pave the way to improve compactness of the policy representation.
An interesting observation that comes up from the empirical demonstration is that different policies result in different sizes of S AFE Z ONES , and that the optimal policy does not necessarily has the smallest S AFE Z ONE . Acknowledgement
LC is supported in part by the Ariane de Rothschild Women Doctoral Program.
This project has received funding from the European Research Council (ERC) under the European Union’s
Horizon 2020 research and innovation program (grant agreement No. 882396), by the Israel Science
Foundation (grant number 993/17), Tel Aviv University Center for AI and Data Science (TAD), and the Yandex Initiative for Machine Learning at Tel Aviv University.
Acknowledgement
LC was supported in part by the Ariane de Rothschild Women Doctoral Program. This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program (grant agreement No. 882396), by the Israel Science
Foundation (grant number 993/17), Tel Aviv University Center for AI and Data Science (TAD), and the Yandex Initiative for Machine Learning at Tel Aviv University.
A Proofs of SectionA.1 Greedy by Threshold Algorithm
A naive approach to the S AFE Z
ONE problem is to return all states s ∈ S with probability p ( s ) ≥ β , for some parameter β > 0 , see Algorithm 3.
Algorithm
Greedy by Threshold
Parameter: β > 0 , { p ( s ) } s ∈S return { s ∈ S : p ( s ) ≥ β }
Lemma 3.1.
For any ρ, β ∈ (0 , 1) , the G REEDY BY T HRESHOLD
A LGORITHM returns a set that is ( ρ + k ∗ β, Hβ ) − S AFE Z ONE .
In particular, for β = ρk ∗ , this set is (cid:16) 2 ρ, k ∗ Hρ (cid:17) − S AFE Z ONE .
Proof.
There are at most Hβ states with probability p ( s ) ≥ β.
Thus | F | ≤ Hβ .
Denote by F ∗ the optimal ( ρ, k ∗ ) − S AFE Z ONE set.
By law of total probability,
[Equation]
Looking at the R.H.S of the inequality, the left term is smaller than ρ by the definition of S AFE Z ONE .
The right term is equal to the probability to reach a state in F ∗ that its probability is smaller than β , i.e., a state in F ∗ \ F.
Using union bound, this can be bounded by k ∗ β.
Lemma A.1.
For every ρ ∈ (0 , 1 / 2 ) , H ∈ N , there exists an MDP and a minimal integer k such that the MDP has a ( ρ, k ) − S AFE Z ONE , but for β = ρ/k G REEDY BY T HRESHOLD
Algorithm returns
F with escape probability ≤ 2 ρ and of size | F | =
Ω( H/β ) .
Proof.
Fix ρ ∈ (0 , 1) .
For ease of the presentation we will assume that 1 − ρβ is an integer (if not, it should be rounded to the nearest integer).
Define A to contain 1 − ρβ · H states, B to contain k −states, and S = { s 0 } ∪ A ∪ B. Consider the following MDP with states S and starting state s 0 .
The transition function is defined as follows:
• For every i ∈ A , Pr[ s A 1 ,i | s 0 ]
=
β and for every j ∈ [ H − 1] , Pr[ s Aj +1 ,i | s Aj,i ] = 1 .
• For s ∈ B , Pr[ s | s 0 ] = 1 − ρ k −• For s ∈ B , Pr[ s | s ] =
The MDP is illustrated in Figure 1.
Clearly, { s 0 } ∪ B is a ( ρ, k ) − S AFE Z ONE .
In addition, G REEDY
BY T HRESHOLD A LGORITHM returns the set of all states, as for every state s ∈
A we have that p ( s ) =
β , p ( s 0 ) = 1 > ρ ≥ β , and for every s ∈ B we have that p ( s )
= 1 − ρ k − 1 > ρk = β .
Thus the size of the returned set is S , which is of size Ω
( H/β ) , which completes the proof.
A.2 Simulation Algorithm
Lemma 3.2.
Fix ρ, β ∈ (0 , 1) .
With probability at least 0 .
99 , S IMULATION
Algorithm returns a set that is (cid:16) ρ + k ∗ β, O ( k ∗ + ρH ln k ∗ β ) (cid:17) − S AFE Z ONE .
In particular, for β = ρk ∗ , this set is (2 ρ, O ( k ∗ H ln k ∗ ))
− S AFE Z ONE .
Proof.
Denote by F ∗ the optimal ( ρ, k ∗ ) − S AFE Z ONE set.
By the law of total expectation, we can split E [ | F | ] into two parts, depending on whether trajectories are entirely in F ∗ or not:
• Trajectories that are entirely in F ∗ contribute at most k ∗ states to F .
[Equation]
F ← { s 0 } for i = 1 . . .
m do τ ← choose a random trajectory
F ← F ∪ τ end for return F
• A trajectory that is not contained in F ∗ contributes at most H states to F .
Thus,
[Equation]
We use Markov’s inequality to get the desired bound on | F | .
For the safety, we first denote the set of all states in F ∗ with probability at least β as Γ = { s ∈ F ∗ | p ( s ) ≥ β } .
We will show that with probability at least 0 .
9995 , it holds that Γ ⊆ F, which will prove our claim, similarly to Lemma 3.1.
For a fixed state s ∈ Γ , the probability that s / ∈ F is bounded by (1 − p (
[Equation]
k ∗ .
Using union bound, the probability that there is a state s ∈ Γ which is not in F is bounded by k ∗ · 0 .
005 k ∗ = 0 .
In other words, with probability at least 0 .
995 , Γ ⊆ F , thus implementing the greedy approach in Algorithm 3 and proving that the probability that a random trajectory escapes F is bounded by ρ + k ∗ β.
Lemma A.2.
For every ρ, γ ∈ (0 , 1) , H, k ∈ N , and β = ρk , there is an integer r ∈ N and MDP with ( ρ, k ) − S AFE Z ONE , but with probability ≥ 1 − γ , S IMULATION
algorithm returns F of size E [ | F | ] ≥ kH ln k with escape probability ∆( F )
=
O ( ρ ) .
Proof.
Fix ρ, γ ∈ (0 , 1) .
Recall that m = 1 β ln k ∗ 0 .
005 and take r = ⌈ m 2 γ ⌉ .
Define A to contain rH states, B to contain k − 1 states, and S = { s 0 } ∪ A
∪ B.
Consider the following MDP with states S and starting state s 0 .
The transition function is defined as follows:
• For every i ∈ A , Pr[ s A 1 ,i | s 0 ]
= ρr and for every j ∈ [ H − 1] , Pr[ s Aj +1 ,i | s Aj,i ] = 1 .
• For s ∈ B , Pr[ s | s 0 ] = 1 − ρ k −• For s ∈ B , Pr[ s | s ] =
The MDP is illustrated in Figure 2.
The set B ∪ { s 0 } is ρ − safe with k states.
We will show that:
• After adding ≥ 1 β ln k
= kρ ln k random trajectories, with probability ≥ 1 − γ
we have that | F | ≥ kH ln
[Equation]
• After adding m random trajectories, we have that with high probability F ∗ ⊆ F , thus ∆( F ) ≤ Ω( ρ ) .
To prove the first property, we claim that with probability ≥ 1 − γ , every time we add a trajectory τ such that τ ∩ A ̸ = ∅ , we add H new states.
Notice that if we ignore s 0 , trajectories in A are entirely unconnected, and each trajectory is chosen randomly with probability
Pr[ s A 1 ,i | s 0 ]
= ρr .
This yields that if s A 1 ,i / ∈ F , then s Aj,i / ∈ F for every j ∈ [ H ] .
As a result, every time we add a new s A 1 ,i to F , we add H − 1 more states to F .
Let N denotes the amount of trajectories sampled with states from A .
The probability that their intersection contains only s 0 is
[Equation]
From the structure of the MDP, we have that E [ N ] = ρm .
Therefore, with probability ≥ 1 − γ ,
[Equation]
The second property follows from Lemma 3.2. A
Greedy at Each Step
Lemma 3.3.
For any ρ ∈ (0 , 1) , if the MDP is layered, G REEDY AT E ACH S TEP A LGORITHM returns a set that is ( ρH, k ∗ ) − S AFE Z ONE .
Input: ρ > 0 , { p ( s ) } s ∈S
F ← { s 0 } for i = 1 . . .
H do
Sort states in S
i , p ( s 1 i ) ≥ . . .
≥ p ( s |S i | i )
j ∗ ← arg min j ∈ [ |S i | ] (cid:80) jr
=1 p ( s ri ) ≥ 1 − ρ
F ← F ∪ (cid:110) s
1 i , . . .
s j ∗ i (cid:111) end for return F
Proof.
Take a random trajectory τ =
( s 1 , s 2 , . . . ) .
For every s i ∈ τ , the probability that s i / ∈ F is bounded by ρ , thus using union bound, the probability that τ has state s i such that s i / ∈ F is at most ρH .
The construction of F guarantees that F is the minimal subset of states such that for every i , the probability that s i is in the subset is at least 1 − ρ .
Assume by contradiction that | F | > k ∗ .
Then there is a time step i such that Pr[ s i ∈ F ∗ ] < 1 − ρ , which is a contradiction, since Pr[ τ ∈ F ∗ ] ≤ min i Pr[ s i ∈ F ∗ ] .
Lemma A.3.
For any ρ ∈ (0 , 1) , there is an MDP and an integer k such that there is a ( ρ, k ) − S AFE Z ONE , but G REEDY
A T E ACH S
TEP Algorithm returns F with escape probability ∆( F ) ≥ Ω( Hρ ) .
Proof.
Fix ρ ∈ (0 , 1) and take k = 3 H + 1 .
Consider the MDP illustrated in Figure 3.
The set { s 0 } ∪ { s i 1 } i ∪ { s i 2 } i ∪ { s i 3 } i form a ( ρ, 3 H + 1) − S AFE Z ONE .
We will prove by induction that the for every time i ,
• p ( s i 1 ) = 1 − 2 ρ , • p ( s i 2 ) =
p ( s i 3 ) = p ( s i 4 ) =
ρ 2 , and
• For every j ∈ { 5 , . . .
, k + 4 } , p ( s ij )
=
ρ 2 k .
It is easy to see that the two properties hold for i = 1 .
For i > 1 ,
[Equation]
Similarly, p ( s i 3 ) =
ρ 2 .
[Equation]
For every j ∈ { 5 , . . .
, k + 4 } , p
[Equation]
The algorithm might return { s 0 } ∪ { s i 1 } i ∪ { s i 2 } i ∪ { s i 4 } i , i.e., instead of taking ∪ i { s i 3 } i it takes ∪ i { s i 4 } i .
Finally, the observation ∆( { s 0 } ∪ { s i 1 } i ∪ { s i 2 } i ∪ { s i 4 } i ) ≥ ρH 4 completes the proof.
B Proofs of SectionB.1 High-level analysis B.1.1 Algorithm Analysis
We define the event
[Equation]
which states that all our EstSafety Subroutine estimations are accurate.
We show that E holds with high probability using Hoeffding’s inequality.
In most of the analysis we condition on E to hold.
The following theorem is the central component in the proof of the main theorem that follows it.
Theorem B.1.
Given ρ, ϵ, λ ∈ (0 , 1) , F INDING S AFE Z ONE Algorithm returns a subset F ⊆ S such that:
The escape probability is bounded from above by ∆( F ) ≤ 2 ρ + 2 ϵ , with probability 1 − λ .
The expected size of F given E is bounded by E [ | F | | E ] ≤ 2 k ∗ .
The sample complexity of the algorithm is bounded by O (cid:16) k ∗ λϵ 2 ln k ∗ λ + Hk ∗ ρλ (cid:17) , and the running time is bounded by O (cid:16)
Hk ∗ λϵ 2 ln k ∗ λ + H 2 k ∗ ρλ (cid:17) , with probability 1 − λ .
To obtain the main theorem, we run F INDING S AFE Z
ONE Algorithm several times and return the smallest output set, F , see the next section for more details.
Theorem B.2.
(main theorem)
Given ϵ, ρ, δ > 0 , if we run F INDING S AFE Z ONE for Θ( 1 δ ) times and return the smallest output set, F ⊆ S , then with probability ≥ 0 .
The escape probability is bounded by ∆( F ) ≤ 2 ρ
+ 2 ϵ. 2.
The size of F is bounded from above by | F | ≤ (2 + δ ) k ∗ .
The total sample complexity and running time are bounded by O ( k ∗ δ 2 ϵ 2 ln k ∗ δ
+ Hk ∗ ρδ 2 ) , and O ( Hk ∗ δ 2 ϵ 2 ln k ∗ δ
+ H 2 k ∗ ρδ 2 ) , respectively.
B.1.2 Proof Technique
Escape probability set size bounds.
To ease the presentation of the proof, we assume that (cid:98) ∆( F ) =
∆( F ) .
This case is interesting by its own, since if the policy and transition function are known, we can compute ∆( F ) efficiently using dynamic programming.
As a result, event E always holds.
In addition, it is clear that the termination of the algorithm implies that (cid:98) ∆( F ) =
∆( F ) ≤ 2 ρ , thus F is (2 ρ + 2 ϵ ) − safe.
The main challenge is bounding the size of F .
A few notations before we start.
Let F ∗ denote a minimal ρ − safe set (of size k ∗ ).
Consider iteration i inside the while–loop.
The random variable G i is the number of states in F ∗ that are added to F in iteration i and B i is the number of states added to F in iteration i that are not in F ∗ ( G stands for good and B for bad ).
Notice that both G i and B i depend on the current set F .
Notice that the size of the output set is exactly (cid:80) i B i
+ G i and that (cid:80)
i G i ≤ k ∗ .
The main idea of the proof technique is to show that by adding trajectories according to the new distribution Q F , we ensure that, in expectation, there are at least as much good states that are added to F as bad states.
Suppose the trajectory τ was chosen to be added to F ∗ by the algorithm.
If τ ⊆ F ∗ , then G i is equal to new F ( τ ) and B i = 0 .
If τ ̸⊆ F ∗ , then B i ≤ new F ( τ ) .
Summarizing these observations, we have the following bounds
[Equation]
where I [ · ] is the indicator function.
Moreover, a direct consequence of the probability in which τ is added to F is that for any set of trajectories T it holds that
[Equation]
where Z is the normalization factor of Q F .
To bound the size of F , we want to show that the algorithm does not add too many states outside of
F ∗ .
We therefore bound E [ B i ] / E
[ G i ] , where the expectations are over the trajectory τ that is added to F according to Q F .
Applying Equation (1) twice, once with T = { τ | τ ⊆
F ∗ } and once with
T = { τ | τ ̸⊆ F ∗ } , we bound the ratio between B i and G i by
[Equation]
We know that Pr τ [ τ ̸⊆ F ∗ ] is always smaller than ρ , so the numerator is ≤ ρ .
A lower bound for the denominator is
[Equation]
Whenever the algorithm is inside the main loop, the safety is at least Pr τ [ new F ( τ ) ̸ = 0] =
∆( F ) > 2 ρ .
Thus (3) is lower bounded by ρ , and overall (2) is less or equal to 1 , which implies that
E [Equation] B i
[Equation]
E [Equation] G i ]
[Equation]
This completes the proof because we know that the algorithm does not add too many states outside of
F ∗ .
More precisely,
[Equation]
Sample complexity.
To discuss the sample complexity, we drop the assumption that the MC is known to a learner, and uses EstSafety Subroutine to approximate ∆( F ) .
The number of calls to
EstSafety is bounded by the size of the output set, F .
Hence, this part of the sample complexity is bounded by | F | · N | F | and we show that is O
( k ∗ ϵ 2 log k ∗ ) .
Another source of sampling is trajectories sampled for purposes of potentially adding them to F .
Observe that at any iteration the set F has escape probability of at least 2 ρ , and each trajectory that escapes F is accepted with probability at least 1 /H .
This implies a lower bound for the probability that a random trajectory is accepted is 2 ρ/H .
This gives an upper bound of 2 | F | ρ H for the expected sample complexity.
Amplification.
Theorem B.1 shows that if E holds, then the set size, | F | , is bounded in expectation by 2 k ∗ .
As Pr[ E ] ≥ 1 − λ implies, from Markov’s inequality, that the size (2 + δ ) k ∗ with small probability of about δ + λ
= O ( δ ) .
If we want to make sure that the actual size is at most (2 + δ ) k ∗ with high probability, we can repeat the process about Θ (cid:0) 1 δ (cid:1) times and take the smallest size set.
For full proofs we refer to Appendix B.
B.2 In-depth Analysis
For convince, we state here Hoeffding’s inequality.
Lemma B.3.
[Hoeffding’s Inequality] Let y 1 , . . .
, y N be independent random variables such that y i ∈
[ a, b ] for every y i with probability 1 .
Then, for any ϵ > 0 ,
[Equation]
B.2.1 Proof of Theorem B.
In this section we provide a complete proof for Theorem B.1.
Throughout the section, we define a few terms and notions.
We will start with proving guarantees regarding a single iteration of the while–loop.
Recall that F ∗ denotes a minimal ρ − safe set (of size k ∗ ).
If there are multiple optimal solutions, choose one arbitrarily.
For the convince of analysis, we denote the values of the algorithm variables at the end of each iteration i of the while–loop by τ i , F i , accept i .
Let j ( i ) denote the value of variable j during the i − th call to EstSafety Subroutine.
In addition, let N i denote the number of trajectories sampled for the j − th time of calling Subroutine EstSafety , i.e., N i = 12 ϵ 2 ln
λ j
( i ) for j ( i ) ≤ i .
For ease of presentation, we recall some of the definitions from the proof technique description.
We say that a trajectory τ is good if all the states in τ are in F ∗ and bad if it escapes it.
I.e., a trajectory is good if τ ⊆ F ∗ and bad if τ ̸⊆ F ∗ .
Additionally, we say that a state s ∈ S is good if it is in F ∗ and bad otherwise.
Namely, a state s is good if s ∈ F ∗ and bad if s / ∈ F ∗ .
Let G i
( F i − 1 ) and B i
( F i − 1 ) be the number of good and bad states added to F i − 1 in iteration i , respectively (notice that G i ( F i − 1 ) and B i ( F i − 1 ) are random variables that depends on F i − 1 ).
For short, whenever it is clear from the context, we write G i and B i respectively.
The following lemma bounds the error in approximating the escape probability.
Lemma B.4.
Let F i − 1 ⊆
S be a subset of of states and ϵ, λ j > 0 be some parameters.
Let S i be a sample of N i ≥ 12 ϵ 2 ln 2 λ j ( i ) i.i.d.
random trajectories.
Then,
[Equation]
Also, as λ j =
[Equation]
Where the last probability is over all the samples S
i made by EstSafety Subroutine.
Proof.
The first part follows directly from Hoeffding’s inequality by taking y i
=
I [ τ ̸⊆ F ] .
Assigning λ j = 3 λ 2( πj ) 2 and applying union bound, we get
[Equation]
The inequality marked by ( ∗ ) follows from the fact that ∆( F ) is estimated once for every time j increases.
We define the event that EstSafety always provides good estimations by
[Equation]
By the above we have that Pr[ E ] ≥ 1 − λ/ 4 .
In the following lemma we assume that if the current escape probability is at least 2 ρ , then the fraction of bad trajectories that escape F i − 1 is bounded from above by the fraction of good trajectories that escape F i − 1 .
Lemma B.5.
Let ρ > 0 and assume that ∆( F i − 1 ) ≥ 2 ρ .
Then,
[Equation]
where the probabilities are over random trajectories.
Proof.
To prove the lemma, we will bound the probability Pr τ [ new F i − 1 ( τ ) ̸ =
∧ τ ̸⊆ F ∗ ] from above and the probability Pr τ [ new F i − 1 ( τ ) ̸ =
∧ τ ⊆ F ∗ ] from below.
Since ∆( F ∗ ) ≤ ρ ,
[Equation]
The assumption ∆( F i − 1 ) ≥ 2 ρ implies that
[Equation]
hence
[Equation]
Putting (5) and (6) together yields the statement.
Now, as long as the algorithm is inside the while–loop (i.e., the escape probability holds (cid:98) ∆( F ) > 2 ρ + ϵ ), it follows that ∆( F ) ≥ 2 ρ with high probability from Lemma B.4.
Combining it with
Lemma B.5 would yield that with high probability over a random trajectory, if the trajectory escapes
F then in expectation it is at least as likely to be good as it is to be bad.
We move on to show the main ingredient of the proof, namely that for any iteration, with high probability, the expected number of good states added to the current set F is larger or equal to the expected number of bad states.
For every iteration i in which we sample τ i both G i and B i depends on the following:
The realizations of the sampled trajectory, τ i , and in particular on new F i − 1 ( τ i ) .
The probability of adding it to F , i.e., 1 /new F i − 1 ( τ i ) .
Next, we prove Equation (4).
Lemma B.6.
Assume event E holds.
Thus, for all iterations i inside the while–loop we have
[Equation]
where the expectation is over the trajectory τ that is sampled from the MC dynamics and added to
F i − 1 according to Q F i − 1 .
Proof.
Since event E holds, we have that ∆( F i − 1 ) ≥ 2 ρ as long as we do not terminate in iteration i .
We can use it to bound E τ [ B i | F i − 1 ] by
E τ
[ B i | F i − 1 ] ≤
H (cid:88)
[Equation]
Lemma B.
[-Pr τ [ new F i − 1 ( τ ) ̸ = 
-]
[Equation]
Theorem B.1.
Given ρ, ϵ, λ ∈ (0 , 1) , F INDING S AFE Z ONE Algorithm returns a subset F ⊆ S such that:
The escape probability is bounded from above by ∆( F ) ≤ 2 ρ + 2 ϵ , with probability 1 − λ .
The expected size of F given E is bounded by E [ | F | | E ] ≤ 2 k ∗ .
The sample complexity of the algorithm is bounded by O (cid:16) k ∗ λϵ 2 ln k ∗ λ + Hk ∗ ρλ (cid:17) , and the running time is bounded by O (cid:16)
Hk ∗ λϵ 2 ln k ∗ λ + H 2 k ∗ ρλ (cid:17) , with probability 1 − λ .
Proof.
Assume that the event E holds, and recall that
[Equation]
We start with the first clause.
Since event E holds, Lemma B.4 in particular implies that ∆( F ) ≤ 2 ρ + 2 ϵ , hence the first clause holds.
For second clause, we will bound E [ | F | | E ] from above by 2 k ∗ .
Since E holds, we have that ∆( F i − 1 ) ≥ 2 ρ , for every i inside the while–loop, thus Lemma B.6 yields
[Equation]
This implies that
[Equation]
where the last inequality follows from the definition of G i , as (cid:80) i G i ≤ | F ∗ | = k ∗ .
We continue with the third clause of the theorem.
Let M denote the sample complexity of the algorithm, namely M = M F + M E where M F is the expected total number of trajectories sampled within the F INDING S AFE Z ONE Algorithm (without the samples made by EstSafety Subroutine) and
M E is total number of trajectories sampled using EstSafety .
We will bound each term separately.
Since E holds, whenever we are inside the while–loop, ∆( F i ) ≥ 2 ρ , which implies that it takes at most 1 / 2 ρ trajectories in expectation to sample a trajectory that escapes F i , and such trajectory is accepted with probability at least 1 /H .
Thus, from Wald’s identity, it follows that
[Equation]
From Markov’s inequality on the above inequality, with probability at least 1 − λ
[Equation]
Moving on to bound M E .
Since E holds, it follows from Equation (8) and Markov’s inequality that
Pr
[Equation]
If | F | ≤ 8 k ∗ λ , the number of calls for Subroutine EstSafety is also bounded by 8 πk ∗ /λ
(we only call
EstSafety after we added states to F ).
It also implies that
[Equation] [Equation]
[Equation]
∗ ) 2 ≤ λ j for every j ≥ 1 .
Thus, if | F | ≤ 8 k ∗ λ ,
[Equation]
Combining the above with Equation (10), we get
[Equation]
As M = M F + M E , union bound over Equation (7), Equation (9) and Equation (11) implies that with probability ≥ 1 − 3 λ/ 4 >
[Equation]
λ ,
[Equation]
For each trajectory we sample we run in time
O ( H ) , e.g., by using a lookup table for maintaining the current set F .
Consequently, if the event in Equation (12) holds then the running time of the algorithm is bounded by
[Equation]
Overall, all the clauses in the lemma hold with probability ≥ 1 − λ .
B.2.2 Proof of Theorem B.
Theorem B.2.
(main theorem)
Given ϵ, ρ, δ > 0 , if we run F INDING S AFE Z ONE for Θ( 1 δ ) times and return the smallest output set, F ⊆ S , then with probability ≥ 0 .
The escape probability is bounded by ∆( F ) ≤ 2 ρ
+ 2 ϵ. 2.
The size of F is bounded from above by | F | ≤ (2 + δ ) k ∗ .
The total sample complexity and running time are bounded by O ( k ∗ δ 2 ϵ 2 ln k ∗ δ
+ Hk ∗ ρδ 2 ) , and O ( Hk ∗ [Equation]
[Equation] [Equation] k ∗ [Equation]
2 k [Equation] 2 ) , respectively.
Proof.
Assume we run F INDING S AFE Z
ONE Algorithm for m = 2ln300 δ times and denote each algorithm output by F i .
Return the smallest set F = argmin F i | F i | .
It follows from Theorem B.1 that for every λ ∈ (0 , 1) , each F i is of expected size E [ | F i | ] ≤ 2 k ∗ , and is (2 ρ + 2 ϵ ) − safe with probability ≥ 1 − λ.
Choosing λ = 0 .
013 m implies
[Equation]
In addition, from Markov’s inequality it follows that for every δ > 0 ,
[Equation]
From the independence of the algorithm runs, for m = 2ln300 δ ,
[Equation]
Hence
[Equation]
As for the sample complexity, let M i denote the (random) sample complexity of the i − th run, and let us denote
[Equation]
From Theorem B.1, M i > ¯ M with probability < λ .
By taking union bound on the sample complexity bound per one run, we get
[Equation]
Where the last inequality follows from Theorem B.1, and λ = 0 .
013 m .
Assigning m = 2ln300 δ and λ = 0 .
013 m = 0 .
01 δ 6ln300 , we get that with probability ≥ 1 − 0 .
[Equation]
Since the algorithm runs in time O ( H ) for every trajectory sampled, if the sample complexity is bounded by the above term, then the total running time is bounded by O (cid:16) Hk ∗ δ 2 ϵ 2 ln Hk
[Equation]
2 (cid:17) . Finally, from union bound over Equation (13), Equation (14) and Equation (15) all the theorem properties hold with probability ≥ 0 .