On Convergence of Average-Reward Off-Policy Control Algorithms in Weakly Communicating MDPs
Abstract
We show two average-reward off-policy control algorithms, Differential Q-learning (Wan, Naik, &
Sutton 2021a) and RVI Q-learning (Abounadi Bertsekas & Borkar 2001), converge in weakly communicating MDPs.
Weakly communicating MDPs are the most general MDPs that can be solved by a learning algorithm with a single stream of experience.
The original convergence proofs of the two algorithms require that the solution set of the average-reward optimality equation only has one degree of freedom, which is not necessarily true for weakly communicating MDPs.
To the best of our knowledge, our results are the first showing average-reward off-policy control algorithms converge in weakly communicating MDPs.
As a direct extension, we show that average-reward options algorithms for temporal abstraction introduced by Wan, Naik, & Sutton (2021b) converge if the Semi-MDP induced by options is weakly communicating.
1. Introduction
In this paper, we extend the convergence results of two off-policy control algorithms for the averagereward objective from a sub-class of MDPs to the most general class of MDPs that could be solved by algorithms learning from a single stream of experience.
These algorithms learn a policy that achieves the best possible average-reward rate, using data generated by some other policy that the agent may not have control of.
Designing convergent off-policy algorithms for the average-reward objective is challenging.
While there are several off-policy learning algorithms in the literature, the only known convergent algorithms are SSP Q-learning and RVI Q-learning, both by Abounadi,
Bertsekas, & Borkar (2001), the algorithm by Ren & Krogh (2001), and Differential Q-learning by Wan, Naik, & Sutton (2021a).
Others either do not have a convergence theory (Schwartz 1993,
Singh 1994; Bertsekas & Tsitsiklis 1996, Das 1999) or have incorrect proof (Yang 2016, Gosavi 2004).
The algorithm by Ren & Krogh (2001) requires knowledge of properties of the MDP which are not typically known.
The convergence of SSP Q-learning requires knowing a state that is recurrent under all policies.
The convergence of the RVI Q-learning algorithm (Abounadi et al. 2001) was developed for unichain MDPs, which just means that the Markov chain induced by any stationary policy is unichain 2 .
The convergence of Differential Q-learning (Wan et al. 2021a) requires a weaker assumption – the solution set of the average-reward optimality equation (formally defined later in (2)) only has one degree of freedom (all the solutions are different by a constant vector).
This assumption can be satisfied if, for example, all optimal policies are unichain.
It is clear that RVI
Q-learning also converges under this assumption.
It is not rare that the solution set of the averagereward optimality equation has more than one degree of freedom (e.g., the MDP at the bottom of
Figure 1).
In this case, the proofs of RVI Q-learning and Differential Q-learning would not go through.
Technically, this is because both two proofs require that the uniqueness of the solution of the action values up to an additive constant (one degree of freedom) in the average-reward optimality equation, so that there is a unique equalibrium in the ordinary differential equations associated with the two algorithms.
[Figure]
solution set of q in the average-reward optimality equation ((2)) is { q (1 , dashed ) = c − 1 , q (2 , solid ) = c, q (2 , dashed ) = c − 2 , ∀ c ∈ R } and has one degree of freedom.
chain every stationary policy is unichain of q is { q (1 , solid ) =
c − 3 , q (1 , dashed ) =
c − 1 , q (2 , solid ) =
c, q (2 , dashed ) =
c − 2 , ∀ c ∈ R } and has one degree of freedom.
The MDP is weakly communicating.
Down : The solution set of q is has more than one degree of freedom.
Note that both { q (1 , solid ) = 0 , q (1 , dashed ) = − 2 , q (2 , solid ) = − 1 , q (2 , dashed ) = − 1 } and { q (1 , solid ) = 0 , q (1 , dashed ) = − 1 , q (2 , solid ) = 0 , q (2 , dashed ) = − 1 } are solutions of q in (2) and these two solutions are not different by a constant vector.
The MDP is weakly communicating.
A more general class of MDPs, called weakly communicating MDPs, may have more than a single degree of freedom in the solution set of the associated optimality equation.
The definition of these MDPs is also natural: except for a possibly empty set of states that are transient under every policy, all states are reachable from every other state in a finite number of steps with a non-zero probability.
It has been observed that the set of weakly communicating MDPs is the most general set of MDPs such that there exists a learning algorithm that can, using a single stream of experience, guarantee to identify a policy that achieves the optimal average reward rate in the MDP (Barlett & Tewari 2009).
In this paper, we show the convergence of RVI Q-learning and Differential Q-learning in weakly communicating MDPs, without requiring any additional assumptions compared with their original convergence theories.
Two key steps in our proof are 1) showing that the solution sets of the two algorithms are non-empty, closed, bounded, and connected, and 2) showing that 0 is the unique solution for the average-reward optimality equation when all rewards are 0 .
With these two results, we use asynchronous stochastic approximation results by Borkar (2009) to show convergence to the solution A sets.
As a direct extension of the above results, we also show the convergence of two algorithms that extend the Differential Q-learning algorithm to the options framework, introduced by Wan et al.
(2021b), if the Semi-MDP induced by a given MDP and a given set of options is weakly communicating.
2. Preliminaries
Consider a finite Markov decision process, defined by the tuple M .
= ( S , A , R , p ) , where S is a set of states, A is a set of actions, R is a set of rewards, and p : S × R × S
×
A → [0 , 1] is the dynamics of the MDP.
At each time step t , the agent observes the state of the MDP S t ∈ S and chooses an action A t ∈
A using some policy b : A×S → [0 , 1] , then receives from the environment a reward R t +1 ∈ R and the next state S t +1 ∈ S , and so on.
The transition dynamics are defined as p ( s ′ , r | s, a ) .
=
Pr( S t +1 = s ′ , R t +1 =
r | S t = s, A t = a ) for all s, s ′ ∈ S , a ∈ A , and r ∈ R .
Denote the set of stationary Markov policies Π .
The reward rate of a policy π starting from a given start state s can be defined as:
[Equation]
Given an arbitrary MDP, the agent may not even be able to visit all states and would therefore miss the chance of learning, for every state s , a policy that achieves the optimal reward rate sup π r ( π, s ) and the agent can at most learn an optimal policy for a set of states, each of which is reachable from every other state.
Such a set of states is often called communicating .
Formally speaking, we say a set of states communicating , if there exists a policy such that moving from either one state in the set to the other one in the set in a finite number of steps has a positive probability.
If the entire state space of an MDP is communicating, we say the MDP communicating .
Weakly communicating
MDPs generalize over communicating MDPs.
In weakly communicating MDPs, in addition to a closed communicating set of states, there is a possibly empty set of states that are transient under every policy.
For weakly communicating MDPs, there exists a unique optimal reward rate r ∗ , which does not depend on the start state.
We say a policy is optimal if it achieves r ∗ regardless of the start state.
The goal of an off-policy control algorithm is to learn an optimal policy from the stream of experience . . .
, S t , A t , R t +1 , S t +1 , . . .
generated by a behavior policy that is not necessarily the same as the agent’s learned policy.
Both RVI Q-learning and Differential Q-learning achieve this goal by solving ¯ r and q in the optimality equation:
[Equation]
It is known that r ∗ is the unique solution of ¯ r and any greedy policy w.r.t.
any solution of q is an optimal policy.
In addition, shifting any solution of q by any constant vector results in the other solution of q .
Finally, unlike in unichain MDPs, where all solutions of q are different by some constant vector, in weakly communicating MDPs, solutions of q may have multiple degrees of freedom.
That is, if q 1 , q 2 are both solutions of q , it is possible that q 1 ̸ = q
+ ce, ∀ c ∈ R , where e denotes the all-one vector.
3. Convergence Results
In this section, we present convergence theories of Differential Q-learning and RVI Q-learning in weakly communicating MDPs, and theories of the two option extensions of Differential Q-learning in weakly communicating SMDPs.
Empirical results verifying the convergence of the two MDP algorithms are presented in Appendix C.
Differential Q-learning updates a table of estimates Q t : S ×
A → R as follows:
[Equation]
where ν ( t, S t , A t ) is the number of times S t , A t has been visited before time step t , { α ν ( t,S t ,A t ) } is a step-size sequence, and δ t , the temporal-difference (TD) error, is:
[Equation]
where ¯ R t is a scalar estimate of r ∗ , updated by:
[Equation]
and η is a positive constant.
We now present the convergence theory of Differential Q-learning.
We first state the required assumptions, which are also required by the original convergence theory of Differential Q-learning by Wan et al.
(2021a).
Assumption 1 For all n ≥ 0 , α n > 0 , (cid:80) ∞ n =
α n = ∞ , and (cid:80) ∞ n =
α 2 n < ∞ .
Assumption 2 Let [ · ] denote the integer part of ( · ) , for x ∈ (0 , 1) , sup n α [ xn ] α n < ∞ and (cid:80) [ ym ] n =
α n (cid:80) mn =
α n → 1 uniformly in y ∈ [ x, 1] .
Assumption 3 There exists ∆ > 0 such that
[Equation]
∆ , a.s., for all s ∈ S , a ∈ A .
Furthermore, for all x > 0 , let N ( n, x ) =
min (cid:8) m > n : (cid:80) mk = n +1 α k ≥ x (cid:9) , the limit lim n →∞
[Equation]
α k
[Equation]
exists a.s. for all s, s ′ ∈ S , a, a ′ ∈
A .
Theorem 1 If M is communicating and Assumptions 1–3 hold, the Differential Q-learning algorithm (Equations 3–5) converges, almost surely, ¯ R t to r ∗ , Q t to the set of solutions of (2) and
[Equation]
(6) and r ( π t , s ) to r ∗ , for all s ∈ S , where π t is any greedy policy w.r.t.
Q t .
Remark:
If the MDP is weakly communicating, that is, it contains transient states, the agent eventually reaches the closed communicating state and never returns to the transient states.
Elements in Q n that are associated with the closed communicating set converge to a set that depends on the values of Q and ¯ R when the MDP reaches the closed communicating set for the first time.
Other elements in Q n would only be visited for a finite number of times and can not be guaranteed to converge to their correct values by any learning algorithm.
Other conclusions of the theorem remain unchanged.
This observation on weakly communicating MDPs also applies to Theorems 2–4.
The update rules of RVI Q-learning are
[Equation]
where
[Equation]
and f : S ×
A → R satisfies the following assumption.
Assumption 4 1) f is L -Lipschitz, 2) there exists a positive scalar u s.t.
f
( e ) =
u and f ( x + ce )
=
f
( x ) + cu , and 3) f
( cx ) =
cf ( x ) , ∀ c ∈ R .
Theorem 2 If M is communicating and Assumptions 1–4 hold, then the RVI Q-learning algorithm (Equations 7–8) converges, almost surely, ¯ R t to r ∗ , Q t to the set of solutions of (2) and r ∗ =
f ( q ) , (9) and r ( π t , s ) to r ∗ , for all s ∈ S , where π t is any greedy policy w.r.t.
Q t .
4. Conclusions
In this paper, we provide, for the first time, convergence results of off-policy average-reward control algorithms in weakly communicating MDPs, which are known to be the most general class of MDPs in which it is possible that a learning algorithm can guarantee to obtain an optimal policy.
Specifically, we show two existing algorithms, RVI Q-learning and Differential Q-learning, converge in weakly communicating MDPs.
As an extension, in Appendix A, we also showed two off-policy average-reward options learning algorithms converge if the SMDP induced by the options is weakly communicating.
Acknowledgements
The authors were generously supported by DeepMind, Amii, NSERC, and CIFAR.
The authors wish to thank Huizhen Yu and Abhishek Naik for discussing several important related papers and discussing ideas to address the technical challenges in the proof.
Computing resources were provided by Compute Canada.
Appendix A. Learning with Options
If the agent has a set of options, it may choose to execute these options.
Each option o in O has two components: the option’s policy π o : A × S →
[0 , 1] , and the termination probability β o : S → [0 , 1] .
For simplicity, for any s ∈ S , o ∈ O , we use π ( a | s, o ) to denote π o ( a, s ) and β ( s, o ) to denote β o ( s ) .
If the agent executes option o at state s , the option’s policy is followed, until the option terminates.
Let L be the set of all possible lengths of options and ˆ R be the set of all possible cumulative rewards.
Note that L and ˆ R are possibly countably infinite.
Let ˆ p ( s ′ , r, l | s, o ) be, when executing option o starting from state s , the probability of terminating at state s ′ , with cumulative reward r and length l .
Formally, for any s, s ′ ∈ S , o ∈
O , r ∈
ˆ R , l ∈ L , ˆ p can be defined recursively in the following way:
[Equation]
where I is an indicator function.
An MDP M and a set of options
O results in a Semi-MDP (SMDP)
ˆ M .
=
( S , O , L , ˆ R , ˆ p ) .
Given an MDP and a set of options, if the agent chooses options using a meta policy, which is a policy that chooses from options, π : O × S → [0 , 1] and executes these options, we denote the sequence of option transitions by . . .
, ˆ S n , ˆ O n , ˆ R n +1 , ˆ S n +1 , . . . .
For the associated SMDP, the reward rate of π given a start state s can be defined as r C ( π, s ) .
=
lim t →∞ E π [ (cid:80) ti =1 R i | S
= s ] /t or r ( π, s ) .
=
lim n →∞ E π [ (cid:80) ni
=
ˆ R i | ˆ S
= s ] / E π
[ (cid:80) ni
=
ˆ L i | ˆ S
= s ] .
Both limits exist and are equivalent (by Puterman’s (1994)
Propositions 11.4.1 and 11.4.7) under the following assumption:
Assumption 5 For each option o ∈ O , when executing the option, there is a non-zero probability of terminating the option after at most |S| stages, regardless of the state at which this option is initiated.
Proposition 1 Under Assumption 5, the expected value as well as the variance of the execution time and cumulative reward of every option at each state exist and are finite.
We say an SMDP is weakly communicating if the MDP with state space S , action space O , reward space ˆ R , and transition function (cid:80) l ˆ p ( s, r, l | s, o ) is weakly communicating.
Just as in the MDP setting, if the SMDP is weakly communicating, the optimal reward rate ˆ r ∗ .
= sup π ∈ ˆΠ r ( π, s ) , where ˆΠ denotes the set of stationary Markov meta policies, does not depend on the start state s .
In addition, the solutions of q may not be different by a constant vector.
Given an MDP and a set of options, the goal of the off-policy control problem is to find a policy that achieves ˆ r ∗ .
Inter-option
Differential Q-learning achieves this goal by solving the optimality equation for SMDPs (Puterman 1994):
[Equation]
where q and ¯ r denote estimates of the option-value function and the reward rate respectively.
Just as in the MDP setting, ¯ r has ˆ r ∗ as its unique solution, and solutions of q may not be different by a constant vector.
Intra-option Differential-learning finds an optimal policy by solving the intra-option optimality equation.
[Equation]
where
[Equation]
The following proposition shows that the set of solutions of (12) is the same as that of (11).
Proposition 2 Any solution of (11) is also a solution of (12) and vice versa.
Now consider option extensions of Differential Q-learning.
Given an SMDP ˆ M = ( S , O , L , ˆ R , ˆ p ) , inter-option Differential Q-learning maintains estimates of option values, and, inspired by Schweitzer (1971), updates estimates using scaled TD errors:
[Equation]
[Equation]
where ν ( n, ˆ S n , ˆ O n ) is the number of visits to state-option pair ( ˆ S n , ˆ O n ) before stage n , L n ( · , · ) comes from an additional vector of estimates L : S ×O
→ R that approximate the expected lengths of state-option pairs, updated by:
[Equation]
where { β n } is the other step-size sequence.
The TD-error δ n in (14) and (15) is
[Equation]
Theorem 3 If ˆ M is communicating, Assumptions 1–3 and 5 hold, except for using ν ( n, s, o ) instead of ν ( t, s, a ) , and that 0 ≤ β n ≤ 1 , (cid:80) n β n = ∞ , and (cid:80) n β 2 n < ∞ , inter-option Differential
Q-learning (Equations 14-17) converges, almost surely, Q n to the set of solutions of (11) and
[Equation]
[Equation]
¯ R n to ˆ r ∗ , and r ( π n , s ) to ˆ r ∗ where π n is a greedy policy w.r.t.
Q n .
Intra-option Differential Q-learning also maintains estimates of option values.
However, instead of updating the estimates using option transitions, it updates for all options using each action transition ( S t , O t , A t , R t +1 , S t +1 ) .
[Equation]
where { α t } is a step-size sequence, ρ t ( o ) .
=
π ( A t | S t ,o ) π
( A t | S t ,O t ) is the importance sampling ratio, and:
[Equation]
where u Q t ∗ is defined in (13).
Theorem 4 If ˆ M is communicating, Assumptions 1–3 and 5 hold, except for using ν ( t, s, o ) instead of ν ( t, s, a ) , intra-option Differential Q-learning (Equations 19-21) converges, almost surely, Q t to the set of solutions of (11) and (18) , ¯ R t to ˆ r ∗ , and r ( π t , s ) to ˆ r ∗ where π t is a greedy policy w.r.t.
Q t .
Appendix B. Proofs B.1. Proof of Proposition
Note that the execution time of each option o ∈
O is the return of executing this option’s policy in a stochastic shortest path MDP (SSP-MDP, Bertsekas 2007) with state space S + {⊥} ( ⊥ is the “terminal” state of the SSP-MDP), action space A and transition function ˆ p satisfying:
[Equation]
Also, note that by Assumption 5, the option’s policy is a ’proper’ policy (Bertsekas 2007) in the
SSP-MDP.
That is, when using the policy, the MDP reaches the terminal state eventually regardless of the start state.
Because the expected value of every proper policy of an SSP-MDP exists and is finite (Section 2.1 of Bertsekas (2007)), the expected value of the execution time of option
o exists.
The existence of the variance can be shown using similar arguments as those used to show the existence of the expectation in Section 2.1 of Bertsekas (2007).
Similarly, the cumulative reward of each option o is the return of executing o ’s policy in an SSPMDP with state space S + {⊥} , action space A , and transition function ˆ p satisfying:
[Equation]
Again the option’s policy is proper and the expected value of the cumulative reward of option o exists.
So does the variance of the cumulative reward.
B.2. Proof of Proposition
By the definition of ˆ p of the SMDP induced by choosing options in an MDP,
[Equation]
, r | s, a )(1 − β ( s ′ , o )) (cid:88)
˜ s, ˜ r,
˜ l
ˆ p (˜ s, ˜ r − r, ˜ l − 1 | s ′ , o )]
(˜ r − ˜ l ¯ r + max o ′ q ( s ′ , o ′ ))
=
(cid:88) a π ( a | s, o ) (cid:88) s ′ ,r p ( s ′ , r | s, a )
β ( s ′ , o )
( r − ¯ r + max o ′ q
( s ′ , o ′ ))
+ (cid:88) a π ( a | s, o ) (cid:88) s ′ ,r p ( s ′ , r | s, a )(1 − β ( s ′ , o ))
(cid:88)
˜ s, ˜ r,
˜ l
ˆ p (˜ s, ˜ r, ˜ l | s ′ , o )]
(˜ r + r − (˜ l + 1)¯ r + max
o
q
( s ′ , o ′ ))
= (cid:88) a π ( a | s, o ) (cid:88) s ′ ,r p ( s ′ , r | s, a ) β
( s ′ , o )
( r − ¯ r + max o ′ q
( s ′ , o ′ ))
+ (cid:88) a π ( a | s, o ) (cid:88) s ′ ,r p ( s ′ , r | s, a )(1 − β ( s ′ , o ))
  r − ¯ r + (cid:88)
˜ s, ˜ r,
˜ l
ˆ p (˜ s, ˜ r, ˜ l | s ′ , o )(˜ r − ˜
l ¯ r + max o ′ q ( s ′ , o ′ )) 
 =
(cid:88) a π ( a | s, o ) (cid:88)
s ′ ,r p ( s ′ , r | s, a ) β
( s ′ , o )
( r − ¯ r + max o ′ q
( s ′ , o ′ ))
+ (cid:88) a π
[Equation]
Therefore any solution of (11) must be a solution of (12)-(13) and vice versa.
B.3. Proof sketch of Theorem 1-Theorem
In this section, we sketch the proof of Theorems 1-4.
It has been shown that all four algorithms introduced above are special cases of the General RVI Q algorithm (Wan et al.
2021a,b).
They also showed that General RVI Q converges under an assumption that is not satisfied for weakly communicating MDPs/SMDPs.
In order to show convergence for weakly communicating MDPs/SMDPs, we replace this assumption with three weaker assumptions that are satisfied for these MDPs/SMDPs.
All other assumptions are the same as those used by Wan et al.
(2021a,b) and can be verified for all four algorithms using their arguments.
We present General RVI Q and prove its convergence with the three new assumptions in Appendix B.8.
The next step of the proof would be verifying the three new assumptions when casting General RVI Q to each of the four algorithms.
This should be straightforward given our Theorem 5 and Lemma 4.
We defer this part to Appendix B.9.
Given that the three assumptions are verified, we have the conclusion part of the convergence theorem of General RVI Q holds for each of the four algorithms.
The convergence of the reward rates of greedy policies w.r.t.
the action/option-values follows the convergence of these values and is shown in Appendix B.10.
B.4. Characterization of the Solution Set
In this section, we characterize the sets that the algorithms described in the previous section converge to.
This section plays a key role in showing their convergence.
We consider the set of solutions of q in the SMDP optimality equation ((11)) and
[Equation]
(22) where f : S ×
O → R satisfies Assumption 4.
It is clear that (11) generalizes over (2) and (22) generalizes over (6), (9), and (18).
And thus the characterization of Q ∞ applies to the sets that action/option values in the aforementioned algorithms are claimed to converge to in Theorems 1–4.
It is known that if the SMDP is weakly communicating, (11) has ˆ r ∗ as its unique solution of ¯ r .
For q , it has been shown by Schweitzer & Federgruen (1978) (we will refer to this work multiple times and thus we use a shorthand “S&F” for simplicity from now on) in their Theorem 4.2 that the set of solutions of q in (11) is closed, unbounded, connected, and possibly non-convex.
The next theorem characterizes Q ∞ .
Theorem 5 If the SMDP is weakly communicating and Assumption 4 holds, Q ∞ is non-empty, closed, bounded, connected, and possibly non-convex.
Proof
First, Q ∞ is non-empty.
To see this point, note that for any solution of q in (11), q ∗ , q ∗ + ce is also a solution for any c ∈ R and thus there must be a c such that (22) holds because f ( x + ce )
=
f
( x )+ cu for any x, c .
Q ∞ is closed because the set of solutions of q in (11) is closed by S&F, the set of solutions of q in (22) is closed because f is Lipschitz and is thus continuous, and the intersection of two closed sets is closed.
Boundedness
We now show that Q ∞ is bounded.
For any q ∈ Q ∞ , let v ( s ) .
=
max o q ( s, o ) .
Rewrite the optionvalue optimality equation (Equation (11)) using v instead of q , we have,
[Equation]
The above equation is known as the state-value optimality equation.
It is easy to verify that
[Equation]
Using this fact, rewrite Equation (22) using v instead of q , we have,
[Equation]
where
[Equation]
Denote the set of solutions of v in (23) by V .
Denote the set of solutions of v in Equation (23) and
Equation (25) by V ∞ .
If V ∞ is bounded, Q ∞ is also bounded because any q ∈ Q ∞ can be obtained from a solution of v ∈ V ∞ with a linear operation in view of Equation (24).
In order to show boundedness, We will need the following two lemmas to proceed.
These two lemmas are similar to Theorem 4.1 (c) and Theorem 5.1 in S&F, except that 1) the ’ max ’ operates over the set of all optimal policies, instead of the set of all deterministic optimal policies as in
Theorem 4.1 (c), and 2) we consider the set of weakly communicating SMDPs while S&F considers general multi-chain SMDPs.
The proofs are also essentially the same.
For completeness, we provide the proofs for these two lemmas in Sections B.5, B.6.
To formally state the Lemma 1, we will first introduce some definitions.
For any π ∈ ˆ Π , let P π denote the |S| × |S| transition probability matrix under policy π .
That is,
[Equation]
Let P ∞
π be the limiting matrix of P π , which is the Cesaro limit of the sequence { P iπ } ∞ i =1 :
[Equation]
Because S is finite, the Cesaro limit exists and P ∞ π is a stochastic matrix (has row sums equal to 1).
Let l π ( s ) .
= (cid:80) o,s ′ ,r,l π ( o | s )ˆ p ( s ′ , r, l | s, o ) l .
And let the fundamental matrix Z π .
=
( I − P ∞
π
+
P ∞ π ) − 1 =
I + lim γ ↑ 1 (cid:80) ∞ n =1 γ n ( P nπ − P ∞ π ) .
Lemma
If the SMDP is weakly communicating, v is a solution of (23) if and only if
[Equation]
In order to state Lemma 2 formally, we need the following definitions.
Define the Bellman error for a state s ∈ S given a policy π ∈ ˆΠ and some v ∈ R |S| , b v,π ( s ) , as follows:
[Equation]
Define R π as the set of recurrent states for P ∞ π .
That is,
[Equation]
Let ˆ Π ∗ denote the set of optimal meta policies.
Define R ∗ as the set of states that are recurrent under some optimal meta policy:
[Equation]
By Theorem 3.2 (b) in S&F, there exists a policy π ∈ ˆΠ ∗ such that R π
= R ∗ .
For any π ∈ ˆΠ , let n ( π ) be the number of recurrent classes for P ∞ π .
Further, define the least number of recurrent classes induced by any optimal meta policy that induces the set of recurrent states R ∗ :
[Equation]
Denote ˆΠ ∗∗ as the set of policies that have R ∗ as their sets of recurrent states and that have n ∗ recurrent classes.
That is,
[Equation]
Theorem 3.2 (d) by S&F shows that all policies within ˆΠ ∗∗ share the same collection of recurrent classes.
Denote the collection of recurrent classes as { R ∗ α | 1 , 2 , · · · , n ∗ } .
The following lemma shows that the solution set of (23) has n ∗ degrees of freedom
[Equation]
Lemma
If the SMDP is weakly communicating, suppose v and v + x are both solutions of (23) , then there exists n ∗ constants y 1 , y 2 , · · · , y n ∗ such that
[Equation]
For any β ∈ { 1 , 2 , · · · , n ∗ } , note that there exists a policy π ( β ) such that R ∗ β is the only one recurrent class under π ( β ) .
To see this point, note that the SMDP is weakly communicating, and thus we can modify π to obtain a new meta policy such that all states except for those in R ∗ β are transient.
Based on the above observation and Lemma 2, for any v ∈ V , we have for any given β ∈ { 1 , . . .
, n ∗ } , there exists a π ( β ) , such that
[Equation]
The first term max s ∈ R ∗ α , Z π ( β ) b v,π ( β ) ( s ) is a constant given v and π ( β ) .
Therefore we see that, for any other solution v + x of (23), if y β is arbitrarily large then y α , ∀ α = 1 , · · · , n ∗ should also be arbitrarily large.
This would violate the Lipschitz assumption on f .
To see this point, let ˜ f ( v ) .
=
f (˜ r + ˜ P v ) .
Let L be a Lipschitz constant of f .
L is also a Lipschitz constant of ˜ f because ˜ P is a stochastic matrix and is thus a non-expansion.
Choose a v 1 ∈ V ∞ and a ˜ v 2 ∈ V ∞ .
a ˜ v 2 ∈ V ∞ , denote m =
∥ v 1 − ˜ v 2 ∥ .
Choose a sufficiently large c > 0 such that cu > L ∥ v 1 + ce − v 2 ∥ =
L ∥ v 1 + ce − ˜ v 2 − ce ∥ =
Lm , where v 2 .
=
˜ v 2 + ce .
Given this choice of c , using ˜ f ( v 1 ) =
˜
f
( v 2 )
=
ˆ r ∗ and ˜ f
( v 1 + ce ) =
˜ f
( v 1 ) + cu , we have (cid:13)(cid:13)(cid:13) ˜
f ( v 1 + ce ) − ˜ f ( v 2 ) (cid:13)(cid:13)(cid:13) =
(cid:13)(cid:13)(cid:13) ˜
f
( v 1 ) + cu − ˜ f
( v 2 ) (cid:13)(cid:13)(cid:13)
= cu > L ∥ v 1 + ce − v 2 ∥ .
This inequality suggests that ˜ f is not Lipschitz continuous with a Lipschitz constant L and thus violates our assumption.
Because the choice of β is arbitrary, V ∞ is upper bounded.
In addition, because the choice of β is arbitrary, we have for any α ∈ { 1 , . . .
, n ∗ } ,
[Equation]
If y α is chosen to be arbitrarily small then y β should also be arbitrarily small for all β = 1 , · · · , n ∗ but again this is not allowed due to (25) for the same reason as mentioned in the previous paragraph.
Therefore y α , ∀ α ∈ { 1 , . . .
, n ∗ } can not be arbitrarily small.
Thus V ∞ is lower bounded.
Combining the upper bound and lower bound, V ∞ is bounded.
Therefore Q ∞ is also bounded.
Connectedness
We now show that Q ∞ is connected.
To this end, again it is enough to show that V ∞ is connected.
Define a function that takes a v ∈ V as input and produces an element in V ∞ as output.
Specifically, let g : V → V ∞ with g ( v ) =
v + xe , where x is the solution of ˜ f
( v + xe )
=
ˆ r ∗ and ˜ f is defined in (32).
Note that x is unique given v because ˆ r ∗ =
˜ f
( v + xe ) =
˜ f
( v )+ xu and thus x = (ˆ r ∗ − ˜ f
( v ))
/u .
We now show that g is Lipschitz continuous.
Consider any v 1 , v 2 ∈ V .
Let x 1 , x 2 satisfy v 1 + x 1 e = g ( v 1 ) and v 2 + x 2 e = g ( v 2 ) respectively.
Again x 1 , x 2 are unique given v 1 , v 2 .
Note that
[Equation]
Therefore g is Lipschitz continuous with Lipschitz constant 1 + L/u .
Finally, because V is connected and the image of any continuous function on a connected set is connected, g ( V ) is connected.
Note that every point in g ( V ) belongs to V ∞ by definition.
Every point in V ∞ also belongs to g ( V ) .
To see this point, pick any v ∈ V ∞ , we can see that v ∈ V and that g ( v ) = v (note that x = 0 given that v ∈ V ∞ ).
Thus v ∈ g ( V ) .
Therefore V ∞ =
g ( V ) is connected.
Given that V ∞ is connected, Q ∞ should also be connected because Q ∞ is a linear transformation of V ∞ (see Equation 24).
Non-convexity
We now show that both V ∞ and Q ∞ are not necessarily convex.
We will show this point by constructing a counter-example, which involves the communicating MDP shown in the left sub-figure of Figure 2.
The optimal reward rate for the MDP is 0 .
Let f ( q ) =
(cid:80) s,a q ( s, a ) .
Such a choice of f satisfies the assumption on f in Theorem 5.
Then by definition, for any v ∈ V ∞ , we have
[Equation]
For the three-state MDP considered here, ˆ r ∗ = 0 .
[Equation]
Therefore,
[Equation] [Equation]
=
[Equation]
v (1) v (2) v (3) (
[Figure]
[Figure] [Figure]
12 ) (4,4,4) (0,0,0) reward =
[Equation]
[Figure] [Figure]
[Figure] [Figure]
) (−1,1,0)
In addition to the above equality, V ∞ needs to satisfy the state-value optimality equation (Equation (23)).
Therefore for any v ∈ V ∞ ,
[Equation]
, which implies
[Equation] [Equation]
Therefore
[Equation]
Graphically, V ∞ corresponds to the two connected yellow line segments in the right sub-figure in
Figure 2.
From the figure, we see that V ∞ is not convex.
Let s and d denote solid and dashed respectively.
Consider any q ∈ Q ∞ , in view of (22),
[Equation]
In addition to the above equality
[Equation]
Q ∞ needs to satisfy the action-value optimality equation (Equation (2)).
Therefore for any q ∈ Q ∞ ,
[Equation]
which implies
[Equation]
Consider two solutions q
[Equation]
q 2 ∈ Q ∞ defined as follows:
[Equation]
The midpoint of q 1 and q 2 , ¯ q .
= 0 .
5 q 1 + 0 .
5 q 2 , satisfies
[Equation]
Note that
[Equation]
Therefore ¯ q does not satisfy the action-value optimality equation ((2)) and ¯ q ̸∈ Q ∞ .
Thus Q ∞ is not convex.
B.5. Proof of Lemma
Part 1: (23) =
⇒ (28) :
Choose any solution of (23), v ∗ , and choose any π ∈ ˆΠ ∗ .
Using Theorem 3.1 (e) by S&F,
[Equation]
(in weakly communicating SMDPs, the “ L ” set in Theorem 3.1 (e) is O ).
Using the above inequality and Lemma 2.1 by S&F, we have
[Equation]
Because π can be any element in ˆΠ ∗ ,
[Equation]
By Theorem 4.1 (c) in S&F, there exists a deterministic optimal policy π , which is apparently an element of ˆΠ ∗ such that
[Equation]
This result, along with (33) shows that
[Equation]
Part 1 is proven.
Part 2: (28) =
⇒ (23) :
Conversely, if v satisfies (28), define
[Equation]
We first show that ˜ v ≥ v .
For any π ∈ ˆΠ ∗ ,
[Equation]
because of the “ max ”
[Equation]
rearranging terms
[Equation]
by Equation 2.2 in S&F
[Equation]
by Theorem 3.1 (a) in S&F
[Equation]
by
[Equation]
We now show that ˜ v ≤ v , which, together with ˜ v ≥ v , implies ˜ v = v .
Let π be a deterministic policy achieving all |S| maxima in (34)
[Equation]
Then we have
[Equation]
Multiplying both sides by P ∞ π ≥ 0 , we have
[Equation]
The l.h.s. is 0 because P ∞ π P π = P ∞
π .
The r.h.s.
≤ 0 because
[Equation]
where we use ˆ r ∗ ≥ r ( π ) , ∀ π ∈ ˆΠ
Combining the above two inequalities, we have
[Equation]
Now we need to use the following lemma, which is essentially the same as Lemma 2.1 by S&F, except that the signs of inequalities are reversed.
The proof follows the same arguments as those used in the proof of Lemma 2.1 by S&F.
Lemma 3 Fix any policy π ∈ ˆΠ , suppose that P ∞ π b = 0 and ( I − P π ) x − b ≤ 0 , then ( I − P ∞ π ) x −
Z π b ≤ 0 .
Let x = v , and b = r π − l π ˆ r ∗ , we see that P ∞ π b = 0 because of (37) and ( I − P π ) x − b ≤ 0 because of (36).
Using the above lemma, we have
[Equation]
Inserting this inequality to (35), we have
[Equation]
Equation
[Equation]
Theorem
[Equation]
because v satisfies (28) .
Combining ˜ v ≥ v and ˜ v ≤ v
[Equation]
we have ˜ v = v and therefore v ( s ) =
˜ v ( s ) =
max o (cid:80) s ′ ,r,
l ˆ p ( s ′ , r, l | s, o )( r − l ˆ r ∗ + v ( s ′ )) , ∀ s ∈ S .
And thus v is a solution of (23).
B.6. Proof of Lemma
Choose π ∈ ˆΠ ∗∗ .
Because v, v + x are both solutions of (23), using part (e)(2) of Theorem 3.1 by
S&F, we have, ∀ s ∈ R ∗ ,
[Equation]
Subtracting, we have x ( s ) =
[ P π x ]( s ) .
Iteratively applying this equation, we have
[Equation]
where d απ is the unique stationary distribution of the α -th recurrent class of P π .
This proves (29).
Because v + x is a solution of (23), by Lemma 1, ∀ s ∈ S ,
[Equation]
by Equation 2.2 in S&F
[Equation]
Using the above equality and (29), (30) and (31) hold.
B.7. Uniqueness of the Zero Solution
The other result we will need to use to show the convergence of the four algorithms introduced in the previous section is the following one.
With this result, the stability of the algorithms can be established using the result by Borkar and Meyn (2000) (see also, Section 3.2 by Borkar 2009).
Lemma
If an SMDP is weakly communicating and all rewards are 0 , 0 is the only element in
Q ∞ .
Proof Given a weakly communicating SMDP, by Lemma 1, any solution of the state-value optimality equation ((23))
satisfies
[Equation]
where R ∗ is defined right after Lemma 1.
Also, because all rewards are 0 , R ∗ is the closed communicating class and ˆΠ ∗ =
ˆΠ contains all stationary policies.
Pick an arbitrary v ∗ ∈ V and an arbitrary policy π ∈ ˆΠ .
For each recurrent class C under P π , we have, by Lemma 1, ∀ s ∈ C , v ∗ ( s ) ≥ (cid:80) s
′ ∈
C d Cπ ( s ′ ) v ∗ ( s ′ ) , where d Cπ denotes the stationary distribution of π in the recurrent class C .
The r.h.s.
only involves class C because starting from a state s ∈ C the MDP can not leave C .
Because the choice of s is arbitrary, min s ∈ C v ∗ ( s ) ≥ (cid:80) s
′ ∈
C d Cπ ( s ′ ) v ∗ ( s ′ )
≥ min s ∈ C v ∗ ( s ) .
Thus (cid:80) s
′ ∈
C d Cπ ( s ′ ) v ∗ ( s ′ )
=
min s ∈ C v ∗ ( s ) .
In addition, because d Cπ ( s ) > 0 for all s ∈ C , v ∗ ( s ′ )
= v ( s ) , ∀ s, s ′ ∈ C .
Now for any s, s ′ ∈ R ∗ , there must exist a π ∈ ˆΠ ∗ =
ˆΠ such that there is a path from s to s ′ and a path from s ′ to s , because s, s ′ are in the same communicating class.
Therefore s, s ′ are in the same recurrent class under P π .
Thus we conclude that v ∗ ( s )
= v ∗ ( s ′ ) .
Therefore ∀ s, s ′ ∈ R ∗ , v ∗ ( s ) =
v ∗ ( s ′ ) .
The transient states values are uniquely determined by values of states in R ∗ .
And in this case they are all equal to the values of states in the communicating class because all rewards are zero.
Thus the solution set of v in the state-value optimality equation ((23)) is { ce : ∀ c ∈ R } .
Now consider the solution set of the option-value optimality equation ((11)).
Let v ( s ) =
max o q ( s, o ) , then (11) transforms to the state-value optimality equation.
Therefore for any two solutions of q in (11), q 1 and q 2 , max o q 1 ( · , o )
=
max o q 2 ( · , o ) + ce for some c .
Furthermore, let q ∗ be any solution of q , q ∗ ( s, o 1 )
=
q ∗ ( s, o 2 ) , ∀ s ∈ S , o 1 , o 2 ∈ O because ∀ s ∈ S , o ∈ O :
[Equation]
Thus the solution set of q in the option-value optimality equation ((11)) is also { ce : ∀ c ∈ R } .
Given (22), cu = cf ( e ) = f ( ce ) =
ˆ r ∗ = 0 and u > 0 implies that c = 0 .
Therefore 0 is the unique solution of q .
The lemma is proved.
B.8. General RVI Q
We now start to present the General RVI Q algorithm.
Let I .
= { 1 , 2 , · · · , k } where k is a positive integer.
Consider solving ¯ r ∈ R and q ∈ R |I| in following equation
[Equation]
[Equation]
where r ∈ R |I| is any fixed |I| -dim vector, and g : R |I| → R |I| satisfies Assumption 6.
We now consider an algorithm solving (38).
This algorithm maintains an |I| -dim vector of estimates
Q ∈ R |I| , and updates Q using
[Equation]
where 1.
{ Y n } is the “update schedule” – it is a set-valued process taking values in the set of nonempty subsets of I with the interpretation: Y n = { i : i th component of Q was updated at time n } , 2.
ν ( n, i ) .
= (cid:80) nk
=
I { i ∈ Y k } , where I is the indicator function (i.e., ν ( n, i ) =
the number of times the i component was updated up to step n )
{ α n } is a step-size sequence, 4.
{ R n } is a sequence of i.i
d. random vectors satisfying E
[ R n ] = r, ∀ n = 0 , 1 , . .
for any Q ∈ R |I| , { G n ( Q ) } is a sequence of i.i
d. random vectors satisfying E [ G n ( Q )( i )]
= g ( Q )( i ) , ∀ i ∈ I , n = 0 , 1 , . .
for any Q ∈ R |I| , { F n ( Q ) } is a sequence of i.i.d.
random
variables
satisfying
E [
F n ( Q
] = f
( Q )
, ∀
n =
where
f : I → R is a function satisfying Assumption 4,
ϵ n
is a sequence
of
random vectors
of size
|I| .
We need the following assumptions in addition to Assumptions 1–4.
Assumption 6 1) g is a max-norm non-expansion, 2) g is a span-norm non-expansion, 3)
g
( x + ce )
= g ( x ) + ce for any c ∈ R , x ∈ R |I| , 4) g ( cx )
= cg
( x ) for any c ∈ R , x ∈ R |I| .
Let
[Equation]
Let F n .
= σ ( Q 0 , ϵ 0 , M 1 , ϵ 1 , M 2 , . . .
, ϵ n − 1 , M n ) be the increasing family of σ -fields.
By the above construction, { M n +1 } is a martingale difference sequence w.r.t.
F n .
That is E
[ M n +1 | F n ] =a.s., n ≥ 0 .
Assumption 7 For n ∈ { 0 , 1 , 2 , . . .
} , E [ ∥ R n − r ∥ 2 | F n ] ≤ K , E [ ∥ G n ( Q ) − g ( Q ) ∥ 2 | F n ] ≤
K (1+ ∥ Q ∥ 2 ) for any Q ∈ R |I| , and E [ ∥ F n ( Q ) − f ( Q ) e ∥ 2 | F n ] ≤ K (1+ ∥ Q ∥ 2 ) for any Q ∈ R |I| for a suitable constant K > 0 .
We make the following assumption on ϵ n .
Assumption 8 E [ ∥ ϵ n ∥ 2 | F n ] < K (1 + ∥ Q n ∥ 2 ) a.s.. Further, ϵ n converges to 0 a.s..
Assumption 9 Equation 38 has a unique solution of ¯ r .
That is, there exists a pair ¯ r ∈ R , q ∈ R |I| satisfying (38) and if both ¯ r 1 , q 1 and ¯ r 2 , q 2 are solutions (38) , ¯ r 1 = ¯ r 2 .
Denoted the unique solution of ¯ r by r # .
Define Q # to be the set of q ∈ R |I| satisfying (38) and r # = f ( q ) .
Assumption 10 Q # is non-empty, bounded, closed, and connected.
Assumption 11 If r ( i ) = 0 , ∀ i ∈ I , then 0 is the only element in Q # .
Assumption 5 is assumed in Section 2 of Wan et al.
(2021b).
Assumptions 1–8 are the same as
Assumptions A.1-A.7 by Wan et al.
(2021b).
Assumptions 9–11 replace Assumption A.8 by Wan et al.
(2021b).
Theorem 6 Under Assumptions 1-11, General RVI Q ( (39) ) converges, almost surely, Q n to Q # and f ( Q n ) to r # .
Proof The proof would by a large degree repeats that of Theorem A.1 by Wan et al.
For simplicity, we only highlight modifications.
Both our proof and the proof of Theorem A.1 study two ordinary differential equations (ODEs):
[Equation] [Equation]
43) where
[Equation]
Lemmas A.1 and A.3 by Wan et al.
and their proofs still hold (by replacing r ∞ with r # ).
Lemma A.2 by Wan et al.
(2021) is replaced by the following one.
The proof follows the same arguments as those for Lemma A.2.
Lemma
The set of equilibrium points of (43) is Q # .
Lemma A.4 by Wan et al.
(2021) is replaced with the following two lemmas.
Lemma 6 If r ( i ) = 0 , ∀ i ∈ I , 0 is the globally asymptotically stable equilibrium for (43) .
Proof We have assumed that 0 is the only element in Q # in Assumption 11 and 0 is thus the unique equilibrium of (43).
The rest follows the proof of Lemma A.4 by Wan et al.
(2021) by replacing q ∞ with 0 .
Lemma 7 Q # is a compact connected internally chain transitive invariant set for the ODE (43) .
Furthermore, any set that contains points not in Q # is not a compact connected internally chain transitive invariant set for the ODE (43) .
Proof According to Assumption 10, Q # is closed and bounded and is thus compact.
Assumption 10 also assumes that Q # is connected.
Q # is an internally chain transitive invariant set for the ODE (43) because every element in Q # is an equilibrium point of the ODE and Q # is connected.
If a set contains a point x ∈ R |I| that is not in Q # , this set can not be internally chain transitive because x is ”transient” and the trajectory of (43) can not be arbitrarily close to x at some arbitrarily large time step.
The last step is to show convergence of a synchronous version of (39)
[Equation]
[Equation]
This result, together with Assumptions 2 and 3, guarantees convergence of the asynchronous update ((44)) by applying results from Section 7.4 by Borkar (2009)
Lemma 8 Equation 44 converges a.s. Q n to Q # as n → ∞ .
Proof The proof essentially follows that of Lemma A.5 with two changes.
First, using Lemma 6 we can show that the ODE ˙ x t = h ∞ ( x t ) =
g ( x t ) − f ( x t )
e − x t has the origin as the unique globally asymptotically stable equilibrium.
Second, the proof of Lemma A.5 uses Borkar’s (2009) Theorem 2, which proves that Q n converges to a (possibly sample path dependent) compact connected internally chain transitive invariant set of ˙ x t = h ( x t ) where
[Equation]
Lemma 7 and Theorem 2 by Borkar (2009) together imply that Q n must converge to Q # .
B.9. Verifying Assumption 9–Assumption
When casting General RVI Q to RVI Q-learning, (38) becomes (2).
It is then clear that Assumptionsatisfies because the associated MDP is communicating, Assumption 10 holds because of Theoremand Assumption 11 holds because of Lemma 4.
When casting General RVI Q to Differential Q-learning, (38) becomes (2) for an MDP with the same transition dynamics as the original one and all rewards being shifted by a constant that depends on initial action-value estimate Q 0 and reward rate estimate ¯ R 0 .
It is clear that this new MDP, just like the original one, is communicating.
Therefore Theorem 5 and Lemma 4 hold and thus
Assumption 9–Assumption 11 hold.
When casting General RVI Q to inter-option Differential Q-learning, (38) becomes (11) for an
SMDP with the same transition dynamics as the original one and the reward of each state-option pair
A being shifted in proportional to its expected duration.
Again the resulting SMDP is communicating and therefore Theorem 5 and Lemma 4 hold and thus Assumption 9–Assumption 11 hold.
When casting General RVI Q to intra-option Differential Q-learning, (38) becomes (12), which is the same as (11) by Proposition 2 for the reward shifted SMDP introduced in the previous paragraph.
B.10. Convergence of Reward Rates of Greedy Policies
Lemma 9 Assume that the SMDP is weakly communicating, suppose Q n converges to Q ∞ almost surely, let π n be a greedy policy w.r.t.
Q n , r ( π n , s )
→ ˆ r ∗ almost surely.
Proof We will need the following lemma for the proof.
Lemma
For any m ∈ { 1 , 2 , 3 , . . .
} , a, b ∈ R m , b > 0 , and for any p ∈ R m such that p ≥ 0 ,
[Equation]
Proof For any s ∈ { 1 , 2 , 3 , . . .
, m } a ( s ) b ( s ) ≥ min s ′ a ( s ′
[Equation] [Equation]
Therefore p
[Equation]
By our assumptions on b and p , p ⊤ b > 0 , we have
[Equation]
p ⊤ a p ⊤ b ≤ max s a ( s ) b ( s ) can be shown in the same way.
Given that Q n converges to Q ∞ , consider r ( π n , s ) where π n is a greedy policy w.r.t.
Q n .
We show that r ( π n , s ) converges to ˆ r ∗ for all s ∈ S .
For any π ∈ ˆΠ , let P π denote the |S| × |O| × |S| × |O| transition probability matrix under policy π .
That is,
[Equation]
Let P ∞
π be the limiting matrix of P π , which is the Cesaro limit of the sequence { P iπ } ∞ i
[Equation]
[Equation]
Because S is finite, the Cesaro limit exists and P ∞ π is a stochastic matrix (has row sums equal to 1).
Let r ( s, o ) .
= (cid:80) s ′ ,r,
l ˆ p ( s ′ , r, l | s, o )
r be the one-stage option reward and l ( s, o ) .
= (cid:80) s ′ ,r,
l ˆ p ( s ′ , r, l | s, o ) l be the one-stage option length.
Let r ( π, s, o ) .
= (cid:80) s ′ ,r,
l ˆ p ( s ′ , r, l | s, o ) r ( π, s ′ ) be the reward rate of policy π starting from s, o .
By part (a) of Theorem 11.4.1 in Puterman (1994),
[Equation]
Thus we have,
[Equation]
where the inequality holds because of Lemma 10, and T Q n ( s, o ) .
=
r ( s, o ) + (cid:80) s ′ ,r,l ˆ p ( s ′ , r, l | s, o ) max o ′ Q n ( s ′ , o ′ ) .
Because the SMDP is weakly communicating, consider a deterministic optimal policy π ∈ ˆΠ ∗ , r ( π, s )
=
ˆ r ∗ , ∀ s ∈ S .
Therefore r ( π, s, o )
=
ˆ r ∗ .
Now we have, for any s, o ,
[Equation]
The first inequality holds because of Lemma 10 and the second inequality holds because π n is a greedy policy w.r.t.
Q n .
With the above results, and that ˆ r ∗ ≥ r ( π n , s, o ) , we have, for any s, o ,
[Equation]
Therefore,
[Equation]
where sp ( x ) = max i x ( i ) − min
x
( i ) denotes the span of vector x .
Because Q n → Q ∞ a.s., every point q in Q ∞ satisfies sp (( T q − q ) /l )
= 0 because ( T q − q ) /l
= ˆ r ∗ e by (11).
In addition, sp (( T Q n − Q n ) /l ) is a continuous function of Q n , by continuous mapping theorem, sp (( T Q n − Q n ) /l )
→ 0 a.s..
Therefore we conclude that r ( π n , s, o )
→ ˆ r ∗ , ∀ s, o .
By definition, r ( π n , s )
= (cid:80) o
π n ( o | s ) r ( π n , s, o ) .
Therefore r ( π n , s )
→ ˆ r ∗
Appendix C. Empirical Results
In this section, we empirically verify our convergence results of Differential Q-learning and RVI
Q-learning by showing the dynamics of estimated action values of the two algorithms in a communicating MDP and a weakly communicating MDP.
1   p = 0.9, r = −p = 0.
[Equation]
We first consider the communicating MDP shown at the bottom of Figure 1.
For this
MDP, we apply Differential Q-learning with initial action values 0 , initial reward rate estimate − 3 , and η = 1 .
The behavior policy chooses action solid with probability 0 .
8 and action dashed with probability 0 .
2 for both two states.
The stepsize is 0 .
We performed 10 runs for each algorithm.
Each run starts from state 1 and lasts for 1000 steps.
Every 10 steps, we recorded the estimated action values and plotted the higher action value for each state in the figure.
Figure 4(left) shows the evolution of these action values.
In the right panel of the same figure, we also show it using RVI
Q-learning with action values being initialized with 0 and f ( q ) =
q (1 , dashed ) .
A more detailed explanation is provided in the figure’s caption.
It can be seen that for both algorithms, 1) for each run, the estimated action-value function converged to a point in the solution set (the black line segments), and 2) for different runs, the estimated action values generally converged to different points in the solution set.
6 max Q(1, )       max Q(2, )
Diﬀerential Q-learning      6 max Q(1, )       max Q(2, )
RVI Q-learning
We also applied both of the two algorithms, with the same parameter settings and initialization in a weakly communicating MDP (Figure 3), which is just the communicating MDP plus a transient state.
In the transient state, taking both solid and dashed actions stays at the transient state with probability 0 .
The MDP moves to with with
The starting from
0 is always −
The starting state is 0 .
Because the agent could spend different amounts of time in the transient state for different runs, the agent may enter the communicating set, which contains states 1 and 2, with different action values associated with state 0 .
The solution set of Differential Q-learning depends on the action values associated with the transient states when entering the communicating class.
Therefore in the figure, the points that the estimated action-value function converged to, corresponding to different runs, are not in a line.
Nevertheless, the estimated action-value function in all runs converged to the green region, which corresponds to the solution set of the action-value optimality equation.
On the other hand, The solution set of RVI Q-learning with the choice of the reference function f ( q ) =
q (1 , dashed ) does not depend on the action values associated with states in the communicating class when entering the class.
Therefore the solution set did not vary across different runs.
Note that if we chose f ( q ) =
q (0 , dashed ) , then again the solution set of RVI Q-learning has that dependence.
6 max Q(1, )       max Q(2, )
Diﬀerential Q-learning      6 max Q(1, )       max Q(2, )
RVI Q-learning
Appendix D. Gosavi’s (2004) Convergence Result Is Incorrect
The convergence result of Gosavi’s proposed algorithm is presented in Theorem 2 of his paper.
In the proof of the theorem, they used Borkar’s two-time scale stochastic approximation result to prove the convergence of the proposed algorithm.
Specifically, they argued that their algorithm is a special case of the general class of algorithms considered in Borkar’s result.
As Gosavi quotes, ”Note that the Eqs.
(48) and (49) for SMDPs form a special case of the general class of algorithms (29) and (30) analyzed using the lemma given in Section 5.1.1. ”
However, a closer look at these equations shows that equation (49) is not a special case of equation (30).
Note that because ρ k is a scalar, y k only has one element and thus the f function in equation (30) does not vary across different state-option pairs.
However, this is not true for the f function in equation (49).
It appears to us that there is no simple fix for this issue.