Hitting Times for Continuous-Time Imprecise-Markov Chains
Abstract
We study the problem of characterizing the expected hitting times for a robust generalization of continuous-time Markov chains.
This generalization is based on the theory of imprecise probabilities, and the models with which we work essentially constitute sets of stochastic processes.
Their inferences are tight lower- and upper bounds with respect to variation within these sets.
We consider three distinct types of these models, corresponding to different levels of generality and structural independence assumptions on the constituent processes.
Our main results are twofold; first, we demonstrate that the hitting times for all three types are equivalent.
Moreover, we show that these inferences are described by a straightforward generalization of a well-known linear system of equations that characterizes expected hitting times for traditional timehomogeneous continuous-time Markov chains.
INTRODUCTION
We consider the problem of characterizing the expected hitting times for continuous-time imprecise-Markov chains [Škulj, 2015, Krak et al., 2017, Krak, 2021, Erreygers, 2021].
These are robust , set-valued generalizations of (traditional)
Markov chains [Norris, 1998], based on the theory of imprecise probabilities [Walley, 1991, Augustin et al., 2014].
From a sensitivity-analysis perspective, we may interpret these sets as hedging against model-uncertainties with respect to a model’s numerical parameters and/or structural (independence) assumptions.
The inference problem of hitting times essentially deals with the question of how long it will take the underlying system to reach some particular subset of its states.
This is a common and important problem in such fields as, e.g., reliability analysis, where it can capture the expected time-to-failure of a system; and epidemiology, to model the expected time-untilextinction of an epidemic.
For imprecise-Markov chains then, we are interested in evaluating these quantities in a manner that is robust against, and conservative with respect to, any variation that is compatible with one’s uncertainty about the model specification.
Erreygers [2021] has recently obtained some partial results towards characterizing such inferences, but has not been able to give a complete characterization and has largely studied the finite-time horizon case.
The problem of hitting times for discrete-time imprecise-Markov chains was previously studied by Krak et al.
[2019], Krak [2020].
In this present work, we largely emulate and extend their results to the continuous-time setting.
We will be concerned with three different types of impreciseMarkov chains.
These are all sets of stochastic processes that are in a specific sense compatible with a given set of numerical parameters, but the three types differ in the independence properties of their elements.
In particular, they correspond to (i) a set of ( time- ) homogeneous Markov chains, (ii) a set of (not-necessarily homogeneous)
Markov chains, and (iii) a set of general—not-necessarily homogeneous nor
Markovian—stochastic processes.
It is known (and perhaps not very surprising) that inferences with respect to these three models do not in general agree; see e.g. [Krak, 2021] for a detailed analysis of their differences.
However, our first main result in this work is that the expected hitting time is the same for these three different types of models.
Besides being of theoretical interest, we want to emphasize the power of this result: it means that even if a practitioner using Markov chains would be uncertain whether the system they are studying is truly homogeneous and/or Markovian, relaxing these assumptions would not influence inferences about the hitting times in this sense.
Purely pragmatically, it also means that we can use computational methods tailored to any one of these types of models, to compute these inferences.
Our second main result is that these hitting times are characterized by a generalization of a well-known system of equations that holds for continuous-time homogeneous Markov chains; see Proposition 2 for this linear system.
The remainder of this paper is structured as follows.
In
Section 2 we introduce the basic required concepts that we will use throughout, formalizing the notion of stochastic processes and defining the inference problem of interest.
In
Section 3, we define the various types of imprecise-Markov chains that we use throughout this work.
We spend some effort in Section 4 to study the transition dynamics of these models, from a perspective that is particularly relevant for the inference problem of hitting times.
In Section 5 we explain and sketch the proofs of our main results, and we give a summary in Section 6.
Because we have quite a lot of conceptual material to cover before we can explain our main results, we are not able to fit any real proofs in the main body of this work.
Instead, these—together with a number of technical lemmas—have largely been relegated to the supplementary material.
PRELIMINARIES
Throughout, we consider a fixed, finite state space X with at least two elements.
This set contains all possible values for some abstract underlying process.
An element of X is called a state , and is usually generically denoted as x ∈ X .
We use R , R ≥ 0 , and R > 0 to denote the reals, the nonnegative reals, and the positive reals, respectively.
N denotes the natural numbers without zero, and we let N 0 :
= N ∪{ 0 } .
For any Y ⊆ X , we use R Y to denote the vector space of real-valued functions on Y ; in particular, R X denotes the space of all real functions on X .
We use ∥·∥ to denote the supremum norm on any such space; for any f ∈ R Y we let ∥ f ∥ :=
max {| f ( x ) | : x ∈ Y} .
Throughout, we make extensive use of indicator functions , which are defined for all A ⊆ Y as I A ( x ) :
= 1 if x ∈ A
and
I A ( x ) :
= 0 , otherwise.
We use the shorthand I
y :
= I { y } .
Let 1 denote the function that is identically equal to 1 ; its dimensionality is to be understood from context.
A map M : R Y → R Y is also called an operator , and we denote its evaluation in f ∈ R Y as Mf .
If it holds for all λ ∈ R ≥ 0 that M ( λf ) =
λMf then M is called non-negatively homogeneous .
For any non-negatively homogeneous operator on R Y , we define the induced operator norm ∥ M ∥ := sup {∥ Mf ∥ : f ∈ R Y , ∥ f ∥ = 1 } .
We reserve the symbol I to denote the identity operator on any space; the domain is to be understood from context.
Note that any linear operator is also non-negatively homogeneous.
Moreover, if M is linear it can be represented as an |Y| × |Y| matrix by arbitrarily fixing an ordering on Y .
However, without fixing such an ordering, we simply use M ( x, y ) :
=
M I y ( x ) to denote the entry in the x -row and y -column of such a matrix, for any x, y ∈ Y .
For any f ∈ R Y and x ∈ Y we then have
Mf ( x ) =
(cid:80) y ∈Y M ( x, y ) f
( y ) , so that Mf simply represents the usual matrix-vector product of M with the (column) vector f .
In the sequel, we interchangeably refer to linear operators also as matrices.
We note the well-known equality ∥ M ∥ = max x ∈Y (cid:80) y ∈Y | M ( x, y ) | for the induced matrix norm.
2.1 PROCESSES & MARKOV CHAINS
We now turn to stochastic processes, which are fundamentally the subject of this work.
The typical (measuretheoretic) way to define a stochastic process is simply as a family ( X i )
i ∈I of random variables with index set I .
This index set represents the time domain of the stochastic process.
The random variables are understood to be taken with respect to some underlying probability space (Ω I , F I , P ) , where Ω I is a set of sample paths , which are functions from
I to X representing possible realizations of the evolution of the underlying process through X .
The random variables
X i , i ∈
I are canonically the maps
X i : ω (cid:55)→ ω ( i ) on Ω I .
However, for our purposes it will be more convenient to instead refer to the probability measure P as the stochastic process.
Different processes P may then be taken over the same measurable space (Ω I , F I ) , using the same canonical variables ( X i ) i ∈I for all these processes.
In this work we will use both discrete - and continuoustime stochastic processes, which corresponds to choosing
I = N 0 or I =
R ≥ 0 , respectively.
In both cases we take
F I to be the σ -algebra generated by the cylinder sets; this ensures that all functions that we consider are measurable.
In the discrete-time case, we let Ω N 0 be the set of all functions from N 0 to X .
A discrete-time stochastic process P is then simply a probability measure on (Ω N 0 , F N 0 ) .
Moreover,
P is said to be a Markov chain if it satisfies the (discretetime) Markov property , meaning that
[Equation] [Equation]
P
[Equation]
X
[Equation]
= x n +1 | X n = x n ) , for all x 0 , . . .
, x n +1 ∈ X and n ∈ N 0 .
If, additionally, it holds for all x, y ∈ X and n ∈ N 0 that
P ( X n +1 = y | X n = x ) = P ( X 1 = y | X
= x ) , then P is said to be a ( time- ) homogeneous Markov chain.
We use P N 0 , P M N 0 , and P HM N 0 to denote, respectively, the set of all discrete-time stochastic processes; the set of all discretetime Markov chains; and the set of all discrete-time homogeneous Markov chains.
In the continuous-time case, we let Ω R ≥ 0 be the set of all cadlag functions from R ≥ 0 to X .
A continuoustime stochastic process P is a probability measure on
(Ω R ≥ 0 , F R ≥ 0 ) .
The process P is said to be a Markov chain if it satisfies the (continuous-time)
Markov property,
P
[Equation]
=
[Equation]
| X
[Equation]
= x
[Equation]
, X t n = x t n )
[Equation]
n
[Equation]
x t n ) for all x t 0 , . . .
, x t n +1 ∈ X , t 0 < · · · < t n ≤ t n +1 ∈ R ≥ 0 , and all n ∈ N 0 .
If, additionally, it holds that
[Equation]
for all x, y ∈ X and all t, s ∈ R ≥ 0 with t ≤ s , then
P is said to be a (time-)homogeneous Markov chain.
We use P R ≥ 0 , P M R ≥ 0 , and P HM R ≥ 0 to denote, respectively, the set of all continuous-time stochastic processes; the set of all continuous-time Markov chains; and the set of all continuous-time homogeneous Markov chains.
We refer to [Norris, 1998] for an excellent further introduction to discrete-time and continuous-time Markov chains.
2.2 TRANSITION DYNAMICS
Throughout this work, we make extensive use of operatortheoretic representations of the behavior of stochastic processes, and Markov chains in particular.
The first reason for this is that such operators serve as a way to parameterize Markov chains.
Moreover, they are also useful as a computational tool, since they can often be used to express inferences of interest; see, e.g., Propositions 1 and 2 further on.
We introduce the basic concepts below, and refer to e.g. [Norris, 1998] for details.
A transition matrix T is a linear operator on R X such that, for all x ∈ X , it holds that T ( x, y ) ≥ 0 for all y ∈ X , and (cid:80)
y ∈X T ( x, y )
= 1 .
There is an important and well-known connection between Markov chains and transition matrices; for any discrete-time homogeneous Markov chain P , we can define the corresponding transition matrix P T as
P T ( x, y ) :
= P ( X 1 = y | X
= x ) for all x, y ∈ X .
Since P is a probability measure, we clearly have that P T is a transition matrix.
Conversely, a given transition matrix T uniquely determines a discrete-time homogeneous Markov chain P with P T
= T , up to the specification of the initial distribution P ( X 0 ) .
For this reason, transition matrices are often taken as a crucial parameter to specify (discrete-time, homogeneous)
Markov chains.
Analogously, for a (non-homogeneous) discrete-time
Markov chain P , we might define a family ( P T n ) n ∈ N 0 of time-dependent corresponding transition matrices, with
P T n ( x, y ) :
= P ( X n +1 = y | X n = x ) , for all x, y ∈ X and n ∈ N 0 .
Conversely, any family ( T n ) n ∈ N 0 of transition matrices uniquely determines a discrete-time Markov chain P with P T n = T n for all n ∈ N 0 , again up to the specification of P ( X 0 ) .
In the continuous-time setting, transition matrices are also of great importance.
However, it will be instructive to first introduce rate matrices.
A rate matrix Q is a linear operator on R X such that, for all x ∈ X , it holds that Q ( x, y ) ≥for all y ∈ X with x ̸
= y , and (cid:80)
y ∈X Q ( x, y )
= 0 .
For any rate matrix Q and any t ∈ R ≥ 0 , the matrix exponential e Qt of Qt can be defined as [Van Loan, 2006]
[Equation]
An alternative characterization is as the (unique) solution to the matrix ordinary differential equation [Van Loan, 2006]
[Equation]
with e Q
= I .
[Equation]
For any t, s ∈ R ≥ 0 it holds that e Q ( t + s ) =
e Qt e Qs , and we immediately have e Q
= I .
The family ( e Qt ) t ∈ R ≥is therefore called the semigroup generated by Q , and Q is called the generator of this semigroup.
Moreover, for any rate matrix Q and any t ∈ R ≥ 0 , e Qt is a transition matrix [Norris, 1998, Thm 2.1.2].
Now let us consider a continuous-time homogeneous
Markov chain P , and define the corresponding transition matrix 1 P T t for all t ∈ R ≥ 0 and x, y ∈ X as
P
[Equation]
t (
[Equation] [Equation]
It turns out that there is then a unique rate matrix P Q associated with P such that P T t = e P Qt for all t ∈ R ≥ 0 .
By combining Equations (1) and (2), we can identify P Q as
[Equation]
As before, in the other direction we have that any fixed rate matrix Q uniquely determines a continuous-time homogeneous Markov chain P with P Q = Q , up to the specification of P ( X 0 ) .
For this reason, rate matrices are often used to specify (continuous-time, homogeneous)
Markov chains.
Let us finally consider a (not-necessarily homogeneous) continuous-time Markov chain P .
For any t, s ∈ R ≥ 0 with t ≤ s , we can then define a transition matrix P T st with, for all x, y ∈ X , P T st ( x, y ) :
= P ( X s = y | X t = x ) .
Under appropriate assumptions of differentiability, this induces a family ( P Q t ) t ∈ R ≥ 0 of rate matrices P Q t , as
[Equation]
In the converse direction we might try to reconstruct the transition matrices of P by solving the matrix ordinary differential equation(s)
[Equation]
By comparing with Equation (1), we see that in the special case where P Q s does not depend on s —that is, where P is homogeneous with P Q s = P Q , say—we indeed obtain
P T st = e P Q ( s − t ) .
However, in general the non-autonomous system (4) does not have such a closed-form solution, and we cannot move beyond this implicit characterization.
2.3 HITTING TIMES
We now have all the pieces to introduce the inference problem that is the subject of this work, viz .
the expected hitting times of some non-empty set of states A ⊂ X with respect to a particular stochastic process.
We take this set
A to be fixed for the remainder of this work.
In the discrete-time case, we consider the (extended realvalued) 2 function τ N 0 : Ω N 0 → R ≥ 0 ∪ { + ∞} given by
[Equation]
for all
[Equation]
This captures the number of steps before a process P “hits” any state in A .
The expected hitting time for a discrete-time process P starting in x ∈ X is then defined as
[Equation]
We use E P (cid:2) τ N 0 | X 0 (cid:3) to denote the extended real-valued function on X given by x (cid:55)→ E P (cid:2)
τ N 0 | X
= x (cid:3) .
When dealing with homogeneous Markov chains, this quantity has the following simple characterization:
Proposition 1.
[Norris, 1998, Thm 1.3.5] Let P be a discrete-time homogeneous Markov chain with corresponding transition matrix P T .
Then h := E P (cid:2) τ N 0 | X 0 (cid:3) is the minimal non-negative solution to the linear system
[Equation]
In the continuous-time case, the definition is analogous; we introduce a function τ R ≥ 0 :
Ω R ≥
→ R ≥ 0 ∪ { + ∞} as τ R ≥ 0 ( ω ) := inf (cid:8) t ∈ R ≥ 0 : ω ( t ) ∈
A (cid:9) for all ω ∈ Ω R ≥ 0 .
This function measures the time until a process “hits” any state in A on a given sample path.
The expected hitting time for a continuous-time process P starting in x ∈ X is
[Equation]
We again use E P (cid:2)
τ R ≥
| X 0 (cid:3) to denote the extended-real valued function on X given by x (cid:55)→ E P (cid:2)
τ R ≥
| X
= x (cid:3) .
Also in this case, the characterization for homogeneous
Markov chains is particularly simple:
Proposition 2.
[Norris, 1998, Thm 3.3.3] Let P be a continuous-time homogeneous Markov chain with rate matrix P Q such that P Q ( x, x ) ̸ = 0 for all x ∈ A c .
Then h := E P (cid:2)
τ R ≥
| X 0 (cid:3) is the minimal non-negative solution to I A h =
[Equation]
IMPRECISE-MARKOV CHAINS
Let us now introduce imprecise-Markov chains [Hermans and Škulj, 2014, Škulj, 2015, Krak et al., 2017], which are the stochastic processes that we aim to study in this work.
Their characterization is based on the theory of imprecise probabilities [Walley, 1991, Augustin et al., 2014].
We here adopt the “sensitivity analysis” interpretation of imprecise probabilities.
This means that we represent an imprecise-Markov chain simply as a set P of stochastic processes.
Intuitively, the idea is that we collect in P all (traditional, “precise”) stochastic processes that we deem to plausibly capture the dynamics of the underlying system of interest.
Inferences with respect to P are defined using lower- and upper expectations, given respectively as
[Equation]
So, their inferences represent robust —i.e. conservative— and tight lower- and upper bounds on inferences with respect to all stochastic processes that we deem to be plausible.
3.1 SETS OF PROCESSES & TYPES
We already mentioned that an imprecise-Markov chain is essentially simply a set P of stochastic processes.
Let us now consider how to define such sets.
We start by considering the discrete-time case; then, clearly,
P will be a set of discrete-time processes.
We will parameterize such a set with some non-empty set T of transition matrices.
Our aim is then to include in P all processes that are in some sense “compatible” with T .
However, at this point we are faced with a choice about which type of processes to include in this set, and these different choices lead to different types of imprecise-Markov chains .
Arguably the conceptually most simple model is P HM T , which contains all homogeneous Markov chains P whose corresponding transition matrix is included in T :
[Equation]
However, we could instead consider P M T , which is the set of all (not-necessarily homogeneous)
Markov chains whose time-dependent transition matrices are contained in T :
P
[Equation]
for all n ∈ N
[Equation]
The last choice that we consider here is the set P I T , which essentially contains all discrete-time processes whose singlestep transition dynamics are described by T .
Its characterization is more cumbersome since we have not expressed these general processes in terms of transition matrices, but we can say that it is the set of all P ∈ P N 0 such that for all n ∈ N 0 and all x 0 , . . .
, x n ∈ X , there is some T ∈ T such that for all y ∈ X it holds that
[Equation]
This last type is called an imprecise-Markov chain under epistemic irrelevance , whence the superscript ‘ I ’.
Note that the three types P HM T , P M T , and P I T capture not only “plausible” variation in terms of parameter uncertainty—expressed through the set T —but also variation in terms of the structural independence conditions that we consider!
So, from an applied perspective, if someone is not sure whether the underlying system that they are studying is truly Markovian and/or time-homogeneous, they might choose to use different such sets in their analysis.
In the continuous-time case, we again proceed analogously.
First, we fix a non-empty set Q of rate matrices, which will be the parameter for our models.
We then first consider the set P HM Q of all homogeneous Markov chains whose rate matrix is included in Q :
[Equation]
The other two types are constructed in analogy to the discrete-time case, but unfortunately we don’t have the space for a complete exposition of their characterization.
Instead we refer the interested reader to [Krak et al., 2017,
Krak, 2021] for an in-depth study of these different types and comparisons between them; in what follows we limit ourselves to a largely intuitive specification.
The model P M Q is the set of all continuous-time (notnecessarily homogeneous)
Markov chains whose transition dynamics are compatible with Q at every point in time.
This includes in particular all Markov chains P satisfying the appropriate differentiability assumptions to meaningfully say that the time-dependent rate matrices P Q t —as in Equation (3)—are included in Q for all t ∈ R ≥ 0 .
However, P M Q also contains other processes that are not (everywhere) differentiable; see e.g. [Krak, 2021, Sec 4.6 and 5.2] for the technical details.
The most involved model to explain is again P I Q , which includes all continuous-time processes whose time- and history-dependent transition dynamics can be described using elements of Q .
It includes, but is not limited to, appropriately differentiable processes P such that for all n ∈ N 0 , all t 0 < · · · < t n ∈ R ≥ 0 , and all x t 0 , . . .
, x t n ∈ X , there is some Q ∈ Q such that for all y ∈ X it holds that
[Equation]
We again refer to [Krak, 2021, Sec 4.6 and 5.2] for the technical details involving the additional elements of P I Q that are not appropriately differentiable.
Importantly, we note the nested structure [Krak, 2021, Prop 5.9]
[Equation]
Q ⊆ P I Q , where the inclusions are strict provided Q isn’t trivial.
For notational convenience, we will use identical sub- and superscripts to denote the corresponding lower- and upper expectations for any of these imprecise-Markov chains; e.g., we let E HM T [ · | · ] := E P HM T [ · | · ] .
3.2 IMPRECISE TRANSITION DYNAMICS
Let us now introduce some machinery to describe the dynamics of imprecise-Markov chains.
In particular, we here move from the set-valued parameters T and Q used in Section 3.1, to their dual representations; these are operators that can serve as computational tools.
In Section 3.1, we described discrete-time impreciseMarkov chains using non-empty sets T of transition matrices.
With any such set, we can associate the corresponding lower- and upper transition operators T and T on R X , defined respectively as
[Equation]
for all f ∈ R X .
More generally, any operator T (resp.
T ) on R X is a lower (resp.
upper ) transition operator if for all f, g ∈ R X , all λ ∈
R ≥
0 , and all x ∈ X , it holds that [De Bock, 2017]
min y ∈X f ( y ) ≤
Tf ( x ) and Tf ( x ) ≤ max y ∈X f ( y ) 2.
Tf + Tg ≤ T ( f + g ) and T ( f + g ) ≤
Tf + Tg 3.
T ( λf ) =
λTf and T ( λf ) =
λTf .
It should be noted that lower- and upper transition operators are conjugate, in that any T induces a corresponding upper transition operator T ( · ) = − T ( −· ) , and vice versa .
Moreover, any transition matrix T is also a lower—and, by its linearity, upper—transition operator.
It is easily verified that the lower- and upper transition operators corresponding to a given non-empty set T are, indeed, lower- and upper transition operators.
Conversely, with a given lower transition operator T , we can associate the set of transition matrices that dominate it, in the sense that
T T := (cid:8) T : T a trans.
mat.
Tf ≥ Tf for all f ∈ R X (cid:9) .
This set satisfies the following important properties:
Proposition 3.
[Krak, 2021, Sec 3.4] Let T be a lower transition operator with conjugate upper transition operator
T ( · ) = − T ( −· ) and dominating set of transition matrices T T .
Then T T is a non-empty, closed, and convex set of transition matrices that has separately specified rows,and for all f ∈ R X it holds that Tf = inf T ∈T T Tf and
Tf = sup T ∈T T Tf .
Moreover, for all f ∈ R X there is some
T ∈ T T such that Tf
=
Tf , and there is some—possibly different— T ∈ T T such that Tf =
Tf .
Notably, there is a one-to-one relation between non-empty sets of transition matrices that are closed and convex and have separately specified rows, and lower (or upper) transition operators: if T is the lower transition operator for the set
T , and if T satisfies these properties, then T = T T [Krak, 2021, Cor 3.38].
Hence these objects may serve as dual representations for each other.
One reason that this is important is the use of T as a computational tool; under the conditions of this duality it holds that for any function f ∈ R X and any n ∈ N 0 , we can write [Hermans and Škulj, 2014]
[Equation]
where T is the lower transition operator for T .
This reduces the problem of computing such inferences for the impreciseMarkov chains P M T and P I T to solving n independent linear optimization problems over T ; first compute f 1 :=
Tf , then compute f 2 :=
T f 1 = T 2 f , and so forth.
Note that this method in general only yields a conservative bound on the corresponding inference for P HM T , as the minimizers T k that obtain T k f k − 1 =
Tf k − 1 may be different at each step.
We next consider the dynamics in the continuous-time setting.
We proceed analogously to the above: we first consider a non-empty and bounded 7 set Q of rate matrices.
With this set, we then associate the corresponding lower- and upper rate operators Q and Q on R X , defined as
[Equation]
More generally, any operator Q (resp.
Q ) on R X is a lower (resp.
upper ) rate operator if for all f, g ∈ R X , all λ ∈
R ≥ 0 and µ ∈ R , and all x, y ∈ X with y ̸
= x , it holds that [De Bock, 2017]
Q ( µ 1 )( x ) = 0 and Q ( µ 1 )( x ) =2.
Q I y ( x ) ≥ 0 and Q
I y ( x ) ≥3.
Qf + Qg ≤ Q ( f + g ) and Q ( f + g ) ≤ Qf + Qg 4.
Q ( λf ) =
λQf and Q ( λf ) =
λQf
As before, such objects are conjugate, in that if Q is a lower rate operator, then Q ( · ) = − Q ( −· ) is an upper rate operator.
Moreover, any rate matrix Q is also a lower (and upper) rate operator.
There is again a duality between lower (or upper) rate operators, and sets of rate matrices.
For fixed Q and with the dominating set of rate matrices Q Q defined as
Q Q := (cid:8) Q : Q a rate mat.
, Qf ≥ Q f for all f ∈ R X (cid:9) , we have the following result:
Proposition 4.
[Krak, 2021, Sec 6.2] Let Q be a lower rate operator with conjugate upper rate operator Q ( · ) = − Q ( −· ) and dominating set of rate matrices Q Q .
Then Q Q is a non-empty, compact, and convex set of rate matrices that has separately specified rows, and for all f ∈ R X it holds that Qf = inf
[Equation]
and
[Equation]
Moreover, for all f ∈ R X there is some Q ∈ Q Q such that
Qf = Qf , and there is some—possibly different— Q ∈ Q Q such that Qf = Qf .
Now fix any lower rate operator Q and any t ∈ R ≥ 0 , and let
[Equation]
The operator e Qt is then a lower transition operator [De Bock, 2017], and the family ( e Qt ) t ∈ R ≥ 0 is a semigroup of lower transition operators; it satisfies e Q ( t + s )
=
e
Qt e
Qs for all t, s ∈ R ≥ 0 , and e Q
=
I .
The analogous construction with an upper rate operator Q instead generates a semigroup ( e Qt ) t ∈ R ≥ 0 of upper transition operators.
When
Q and Q are taken with respect to the same set Q , these semigroups satisfy, for all t ∈ R ≥ 0 , f ∈ R X , and Q ∈ Q ,
[Equation]
Here the importance again derives from the use as a computational tool; under the conditions of duality between Q and
Q , we have for any f ∈ R X and any t ∈ R ≥ 0 that [Škulj, 2015, Krak et al
[Equation] [Equation]
Hence such inferences can be numerically computed by approximating (6) with a finite choice of n , and then solving n independent linear optimization problems over Q .
Error bounds for this scheme are available in the literature [Škulj, 2015, Krak et al., 2017, Erreygers, 2021].
3.3 CLASS STRUCTURE
Let us now fix a set Q of rate matrices that we will use in the remainder of this work.
Throughout, let Q and Q denote the lower- and upper rate operators associated with
Q .
We impose several standard regularity conditions on this set: we assume that Q is non-empty, compact, convex, and that it has separately specified rows.
These are common assumptions that are imposed to ensure the duality between
Q and Q , which in turn guarantees that inferences with the induced imprecise-Markov chains remain well-behaved, as well as analytically (and, often, computationally) tractable.
We now have all the pieces to start studying the inference problem that is the subject of this work: the lower- and upper expected hitting times of the set
A ⊂ X for continuous-time imprecise-Markov chains described by Q .
Before we begin, let us impose two additional conditions on the dynamics of the system.
Assumption 1.
We assume that all states in A are absorbing , which is equivalent to requiring that Q ( x, x ) =
for all
Q ∈ Q and all x ∈ A .
Note that this does not influence the inferences in which we are interested, since those only deal with behavior at times before states in A are reached.
However, imposing this explicitly substantially simplifies the analysis.
Next, we assume that the set A is lower reachable from any state x ∈
A c [De Bock, 2017].
This means that we can construct a sequence x 1 , . . .
, x n +1 ∈ X starting in any x 1 ∈
A c and ending in some x n
+1 ∈
A such that, for all k = 1 , . . .
, n , it holds that Q I x k +1 ( x k ) > 0 .
This is equivalent [De Bock, 2017] to
Assumption 2.
We assume e Qt I
A ( x ) >
for all t ∈ R >and all x ∈ A c .
Essentially, this means that for all elements of our impreciseprobabilistic models the probability of eventually hitting A is bounded away from zero.
This ensures that the expected hitting times remain bounded for all P ∈ P I Q , so that we can ignore any extended real-valued analysis.
It also implies that for all Q ∈ Q we have that Q ( x, x ) ̸ = 0 for all x ∈ A c , which is relevant to meet the precondition of Proposition 2.
As a practical point, De Bock [2017] gives an algorithm to check whether a given set Q satisfies this condition.
On a technical level, Assumption 2 is the crucial one for our results, and—unlike with Assumption 1—it cannot really be ignored in practice.
However, based on earlier work by Krak et al.
[2019] in the discrete-time setting, we hope in the future to strengthen our results to hold without this assumption.
SUBSPACE DYNAMICS
In the context of hitting times, the interesting behavior of a process actually occurs before it has reached a target state in A .
Hence it will be useful to introduce some machinery to study the transition dynamics as it relates to the states A c .
To introduce the notation in a general way, choose any nonempty Y ⊂ X .
Then for any f ∈ R X , let f | Y ∈ R Y denote the restriction of f to Y .
Conversely, for any f ∈ R Y , let f ↑ X ∈ R X denote the unique extension of f to X that satisfies f ( x ) =
for all x ∈ X \ Y .
Moreover, for any operator M on R X , we define the operator M | Y on R Y as
[Equation]
for all f ∈ R Y .
This somewhat verbose notation is perhaps most easily understood when M is a linear operator, i.e. a matrix.
In that case, M | Y is simply the |Y| × |Y| sub-matrix of M on the coordinates in Y .
The definition above allows us to extend this notion also to non-linear operators, and to lower- and upper transition and rate operators, specifically.
Now for any rate matrix Q ∈ Q , we call G :=
Q | A c its corresponding subgenerator .
For any t ∈ R ≥ 0 , we then define e Gt :=
e Qt | A c .
We have the following result:
Proposition 5.
Fix Q ∈ Q and let G be its subgenerator.
Then e Gt = lim n → + ∞ (cid:0)
I + t / n G (cid:1) n for all t ∈ R ≥ 0 .
Moreover, the family ( e Gt ) t ∈ R ≥ 0 is a semigroup.
Analogously, we define G :=
Q | A c and G := Q | A c to be the lower- and upper subgenerators corresponding to Q and Q , respectively.
We also let e Gt :=
e Qt | A c and e Gt :=
e Qt | A c .
Perhaps unsurprisingly, we then have:
Proposition 6.
It holds that e Gt = lim n → + ∞ (cid:0)
I + t / n G (cid:1) n and e Gt = lim n → + ∞ (cid:0) I + t / n G (cid:1) n for all t ∈ R ≥ 0 .
Moreover, the families ( e Gt ) t ∈ R ≥ 0 , ( e Gt ) t ∈ R ≥ 0 are semigroups.
Our Assumption 2 implies the norm bound:
Proposition 7.
For any t > 0 , it holds that ∥ e Gt ∥ < 1 .
It is a straightforward consequence of the use of the supremum norm, together with Equation (7) and the fact that e Qt and e Qt are (upper) transition operators, that also ∥ e Gt ∥ ≤ ∥ e Gt ∥ < 1 for all t ∈ R > 0 .
Hence by the semigroup property we immediately have that lim t → + ∞ ∥
e Gt ∥ = 0 .
This also implies the following well-known result.
Proposition 8.
[Taylor and Lay, 1958, Thm IV.1.4]
For any Q ∈ Q with subgenerator G , and all t > 0 , the inverse operator ( I − e Gt ) − 1 exists, and ( I − e Gt ) − 1 = (cid:80) + ∞ k =
e Gtk .
This allows us to characterize hitting times for discrete-time homogeneous Markov chains whose transition matrix is given by e Qt , as follows.
Proposition 9.
Choose any Q ∈ Q , let G be its subgenerator, and fix any ∆ > 0 .
Let P ∈ P
HM N 0 be such that P T =
e Q ∆ .
Then the expected hitting times h := E P [ τ N 0 | X 0 ] satisfy h | A c
= ( I − e G ∆ ) − 1 1 and h ( x ) = 0 for all x ∈ A .
Proof.
By Proposition 1, in x ∈ A c we have that
[Equation]
Conversely, it is immediate from the definition that h ( x ) = 0 for all x ∈ A .
This implies that h
= ( h | A c ) ↑ X , and hence h
[Equation] A [Equation] (cid:0) e
[-| A c = 
-][Equation]
c .
Re-ordering terms we have ( I − e G ∆ ) h | A c = 1 .
Now use
Proposition 8 and multiply with ( I − e G ∆ ) − 1 .
We need the following observation:
Lemma 1. Consider any Q ∈ Q with subgenerator G , and let σ ( G ) be the set of eigenvalues of G .
Then Re λ < 0 for all λ ∈ σ ( G ) .
This implies that 0 / ∈ σ ( G ) , and so we have:
Corollary 1.
For any Q ∈ Q with subgenerator G , the inverse operator G − 1 exists.
This allows us to characterize hitting times for continuoustime homogeneous Markov chains:
Proposition 10.
Choose any Q ∈ Q , let G be its subgenerator, and let P ∈ P HM R ≥ 0 with P Q = Q .
Then the expected hitting times h := E P [ τ R ≥
| X 0 ] satisfy h | A c = − G −
1and h ( x ) =
for all x ∈ A .
Proof.
By Proposition 2, in x ∈ A c we have that
[Equation]
( x ) .
Conversely, it is immediate from the definition that h ( x ) = 0 for all x ∈ A .
This implies h
= ( h | A c ) ↑ X , and hence
[Equation]
Now use Corollary 1 and multiply with G − 1 .
4.1 QUASICONTRACTIVITY OF SUBSPACE DYNAMICS
We already know from Proposition
that ∥ e Gt ∥ < 1 for all t ∈ R > 0 .
Since e G
= I (because it is a semigroup), it follows that ∥ e Gt ∥ ≤ 1 for all t ∈ R ≥ 0 .
A semigroup that satisfies this property is said to be contractive .
Moreover,
Proposition 7 together with the semigroup property implies that lim t → + ∞ ∥
e Gt ∥ = 0 .
A semigroup that satisfies this property is said to be uniformly exponentially stable , and in such a case the following result holds:
Proposition 11.
There are M ≥ 1 and ξ > 0 such that
[Equation]
for all
[Equation]
This result means that the norm ∥ e Gt ∥ decays exponentially as t grows.
However, for technical reasons we require an exponentially decaying norm bound with M = 1 ; if this holds the semigroup is said to be quasicontractive .
It is not clear that obtaining such a bound is possible when ∥ e Gt ∥ is induced by the supremum norm ∥·∥ on R A c .
However, we can get it by defining a different norm ∥·∥ ∗ on R A c .
We then obtain the quasicontractivity with respect to the induced operator norm ∥·∥ ∗ .
Because R A c is finitedimensional these norms are equivalent, and such a result suffices for our purposes.
This re-norming trick is originally due to Feller [1953], and an analogous construction is commonly used for semigroups of linear operators; see e.g. [Renardy and Rogers, 2006, Thm 12.21].
So, consider the ξ > 0 from Proposition 11, and let
[Equation]
where | f | denotes the elementwise-absolute value of f .
Proposition 12.
The map f (cid:55)→ ∥
f ∥ ∗ is a norm on R A c .
Moreover, we have the desired result:
Proposition 13.
We have ∥ e Gt ∥ ∗ ≤ e − ξt for all t ∈ R ≥ 0 .
Finally, the same bound holds for precise models:
Proposition 14.
For any Q ∈ Q with subgenerator G it holds that (cid:13)(cid:13)
e Gt (cid:13)(cid:13) ∗ ≤ e − ξt for all t ∈ R ≥ 0 .
HITTING TIMES AS LIMITS
We now have all the pieces to explain the proof of our main results.
The trick will be to establish a connection between hitting times for continuous-time imprecise-Markov chains, and hitting times for discrete -time imprecise-Markov chains, for which analogous results were previously established by Krak et al.
We essentially just look at a discretized continuous-time
Markov chain taking steps of some fixed size ∆ > 0 , derive the expected hitting time for this discrete-time Markov chain, and then take the limit ∆ → 0 + .
The main difficulty is in establishing that this converges uniformly for all elements in our sets of processes; this is why we went through the trouble of establishing quasicontractivity in Section 4.1.
To start, for any Q ∈ Q and ∆ > 0 , let h Q ∆ be the minimal non-negative solution to the linear system
[Equation]
and let h Q be the minimal non-negative solution to
[Equation]
Then we know from Propositions 1 and 2 that 1 / ∆ h Q ∆ represents the expected hitting times of a discrete-time homogeneous Markov chain with transition matrix e Q ∆ , and that h Q does the same for a continuous-time homogeneous Markov chain with rate matrix Q .
We now have the following result:
Proposition 15.
There are δ > 0 and L > 0 such that
[Equation]
h [Equation] (cid:13)(cid:13) for all ∆ ∈ (0 , δ ) and all Q ∈ Q .
Since (cid:13)(cid:13) h Q (cid:13)(cid:13) is bounded due to Proposition 10:
Corollary 2.
We have lim ∆ → 0 + h Q ∆ = h Q for all Q ∈ Q .
We will now set up the analogous results for impreciseMarkov chains.
First, let
[Equation]
Clearly, it follows from Proposition 2 and the definition of lower- and upper expectations that these quantities represent the lower- and upper expected hitting times for the imprecise-Markov chain P HM Q , i.e. it holds that h [Equation] E [Equation]
Q (cid:2)
τ R[-≥ 
-]
[Equation]
0 | X 0 (cid:3) and h = E HM Q (cid:2)
τ R ≥
| X 0 (cid:3) .
Now for any ∆ > 0 , let h ∆ and h ∆ denote the minimal non-negative solutions to the non-linear systems
[Equation]
12) and
[Equation]
It was previously shown by Krak et al.
[2019] that—up to re-scaling with 1 / ∆ —the quantities h ∆ and h ∆ represent the lower (resp.
upper)
expected hitting times of, identically, the discrete-time imprecise-Markov chains P HM T ∆ , P M T ∆ , and
P I T ∆ parameterized by the set T ∆ of transition matrices that dominate e Q ∆ .
We now set out of prove an analogous result for continuous-time imprecise-Markov chains.
We start with the following:
Proposition 16.
It holds that lim ∆ → 0 + h ∆ = h and lim ∆ → 0 + h ∆ = h .
This property allows us to leverage recent results by Erreygers [2021] and Krak [2021] regarding discrete and finite approximations of lower- and upper expectations in continuous-time imprecise-Markov chains, to obtain our first main result:
Theorem 1.
It holds that
[Equation]
and, moreover, that
[Equation]
Moreover, it follows relatively straightforwardly from
Proposition 16 that the lower- and upper expected hitting times for continuous-time imprecise-Markov chains satisfy an immediate generalization of the system that characterizes the expected hitting times for (precise) continuous-time homogeneous Markov chains.
This is our second main result:
Theorem 2.
Let h and h denote the lower- and upper expected hitting times for any one of P HM Q , P M Q , or P I Q .
Then h is the minimal non-negative solution to the non-linear system
I A h = I A c
+ I A c Q h , and h is the minimal non-negative solution to the non-linear system
I A h = I A c + I A c Q h .
SUMMARY & CONCLUSION
We have investigated the problem of characterizing expected hitting times for continuous-time imprecise-Markov chains.
We have shown that under two relatively mild assumptions on the system’s class structure— viz.
that the target states are absorbing, and can be reached by any non-target state—the corresponding lower (resp.
upper)
expected hitting time is the same for all three types of imprecise-Markov chains.
We have also demonstrated that these lower- and upper expected hitting times h and h satisfy the non-linear systems
[Equation]
in analogy with the precise linear system (5).
Indeed, we conclude that the lower- and upper expected hitting times for any of these three types of imprecise-Markov chains, can be fully characterized as the unique minimal non-negative solutions to these respective systems.
We aim to strengthen these results in future work to hold with fewer assumptions on the system’s class structure.
Acknowledgements
We would like to sincerely thank Jasper De Bock for many stimulating discussions on the subject of imprecise-Markov chains, and for pointing out a technical error in an earlier draft of this work.
We are also grateful for the constructive feedback of three anonymous reviewers.