A Near-Optimal Primal-Dual Method for Off-Policy Learning in CMDP
Abstract
As an important framework for safe Reinforcement Learning, the Constrained
Markov Decision Process (CMDP) has been extensively studied in the recent literature.
However, despite the rich results under various on-policy learning settings, there still lacks some essential understanding of the offline CMDP problems, in terms of both the algorithm design and the information theoretic sample complexity lower bound.
In this paper, we focus on solving the CMDP problems where only offline data are available.
By adopting the concept of the single-policy concentrability coefficient C ∗ , we establish an Ω (cid:16) min {|S||A| , |S| + I } C ∗ ( 1 − γ ) 3 ϵ(cid:17) sample complexity lower bound for the offline CMDP problem, where I stands for the number of constraints.
By introducing a simple but novel deviation control mechanism, we propose a near-optimal primal-dual learning algorithm called DPDL.
This algorithm provably guarantees zero constraint violation and its sample complexity matches the above lower bound except for an ˜ O
((1 − γ ) − 1 ) factor.
Comprehensive discussion on how to deal with the unknown constant C ∗ and the potential asynchronous structure on the offline dataset are also included.
Introduction
Reinforcement Learning (RL) is an important tool for modeling the real world tasks that involve sequential decision making.
Such RL problems are often mathematically described as a Markov
Decision Process (MDP) that maximizes a cumulative sum of rewards.
The safe reinforcement learning, on the other hand, not only cares the reward maximization, but also attempts to ensure a reasonable system performance with respect to certain safety constraints.
Such safety constrained
RL problems are often formulated as the Constrained Markov Decision Process (CMDP) M = ( S , A , P , r, u, γ, ρ 0 ) , where S is a finite state space,
A is a finite action space, γ ∈
(0 , 1) is the discount factor, P ( s ′ | s, a ) stands for the transition probability from s to s ′ under the action a for ∀ ( s, a, s ′ )
∈ S ×A×S , and r : S
×A
→
[ − 1 , 1] is the reward function, ( u i : S ×A → [ − 1 , 1])
i ∈
[ I ] is a set of I utility functions, ρ 0 is the initial state distribution over S .
The goal of CMDP is to find an optimal policy π to maximize the cumulative reward while satisfying a group of constraints:
[Equation]
For the CMDP problem, there has been plenty of on-policy algorithms, see [7, 8, 20, etc.].
However, in real world applications such as training physical robots, where safety is an important measure of performance, the real time on-policy interaction with the environment may suffer from the potential damages to the robots.
Besides, in many non-simulating environments, the on-policy data collection may also be time-consuming.
Therefore, it is crucial to design an off-policy algorithm to solve the CDMP problems, where plenty of historical data are already accumulated while real time interactions are limited.
To our best knowledge, offline CMDP algorithms are rare [12, 27, 29], and the sample complexity guarantees are limited.
In particular, a strong uniform concentrability assumption is required in [12], and the model-based method [27] mainly considers the case an empirical model is known.
Thus it is still not clear how to efficiently solve offline CMDPs with model-free approaches, and there lacks essential understanding of the information theoretic lower bound on the sample complexity of the offline CMDP.
In this paper, we propose a D eviation-controlled Primal-Dual Learning (DPDL) method to solve problem (1).
We adopt the primal-dual strategy developed in [4, 16, 26, 35, etc.]
as the main algorithmic framework while several non-trivial contributions have been made beyond the existing results.
Unlike the aforementioned literatures that exclusively rely on the accessibility of a generative model, DPDL utilizes the offline data, where the distribution shift difficulties of the offline data is tackled by a novel and effective adaptive deviation control mechanism.
If the considered
CMDP instance has a finite (but potentially unknown) concentrability coefficient , DPDL provably finds a policy with O ( ϵ ) -optimal reward and zero constraint violation.
An information theoretical lower bound on the sample complexity of offline CMDP is also derived in this paper, which indicates that our deviation control mechanism achieves a minimax optimal complexity dependence on
I, |S| , |A| , C ∗ .
Main Contribution.
We summarize the contributions in details as follows.
• We propose the DPDL algorithm to solve the CMDP problem (1).
Suppose the CMDP instance satisfies the Slater’s condition and certain prior knowledge on the concentrability coefficient C ∗ is given, DPDL provably finds an ϵ -optimal policy with zero constraint violation using ˜ O (cid:16) min {|S||A| , |S| + I } C ∗ ( [Equation]
[Equation]
(cid:17) offline samples. • We establish an information theoretic sample complexity lower bound of Ω (cid:16) min {|S||A| , |S| + I } C [Equation] for the offline CMDPs, indicating that DPDL is near optimal up to an ˜ O
((1 − γ ) − 1 ) factor.
The necessity of the Slater’s condition for achieving zero constraint violation is also established.
The sharp dependence on the number of constraints is mainly captured by our careful construction of the correlated actions.
• In order to handle the practical situation where C ∗ is unknown, an adaptive version of DPDL is designed with the same sample complexity as DPDL. • Our analysis of DPDL also extends to the asynchronous case, where the offline dataset consists of a sample trajectory generated by certain behavior policy.
In this situation, the sample complexity of DPDL is shown to be ˜ O (cid:16) t 2mix min {|S||A| , |S| + I } C ∗ ( [Table] (cid:17) .
Our handling of the correlated gradient estimators with large variance can also be beneficial to other algorithms under the asynchronous setting.
Related Work.
Recently, considerable efforts have been devoted to the online learning of CMDP.
Under the episodic and tabular setting, several works [7, 8, 20] have achieved the ˜ O (cid:0) (cid:112) |S| 2 |A| T (cid:1) regret and cumulative constraint violation, with different dependence on the episode length H omitted.
Under proper assumptions, zero or bounded cumulative constraint violation can be achieved [1, 17].
In terms of the number of constraints I , MOMA proposed in [34] achieves an ˜ O (cid:0)(cid:112) min {|S| ,I } I |S||A| /T (cid:1) convergence on both average reward gap and constraint violation.
Nevertheless, all the above results adopt the model-based approaches.
Except for [34], they either consider the cases where I = 1 or completely ignore the influence of I in the sample complexity.
Therefore, both deriving an efficient model-free method and obtaining the optimal dependence on I remain open.
Another approach closely related to our paper is the primal-dual method in RL, see [4, 11, 25, 26, 35, etc.].
Given the access to a generative model, the model-free primal-dual method developed in [4] achieves an ˜ O (cid:0) I |S||A| (1 − γ ) 4 ϵ 2 (cid:1) sample complexity to find an ϵ -optimal safe policy.
The deviation control mechanism we develop enables the primal-dual approach to extend beyond the generative model.
Finally, we mention a few related works in the offline RL and safe RL.
Previous offline RL algorithms with sample efficiency guarantees typically assume the uniform concentrability [12, 18, etc.] or lower bounded minimum visitation µ min [32, 33, etc.].
Recently, under the less restrictive assumption of the single-policy concentrability coefficient C ∗ , a minimax optimal sample complexity lower bound of Ω (cid:0) |S| C ∗ (1 − γ ) 3 ϵ 2 (cid:1) for discounted offline MDPs is derived in [21].
A similar Ω (cid:0) H 3 |S| C ∗ ϵ(cid:1) lower bound is also derived for the episodic setting in [28].
Under both settings, offline algorithms with ˜ O ( |S| C ∗ ϵ − 2 ) sample complexity (with different (1 − γ ) − 1 or H factors omitted) have been discovered with either model-based [15, 21, 28, 31] or model-free approaches [22, 30].
In terms of the offline CMDP problem, the only existing results are [12, 27, 29], where [29] only provides asymptotic convergence, [12] relies on a much stronger uniform concentrability assumption, and [27] is a model based method that potentially suffers an O (( C ∗ ) 2 ) dependence.
Compared to these works, our method is model-free and has an optimal O ( C ∗ ) dependence on the concentrability coefficient.
Problem setup 2.1 LP formulation of CMDP problem
For any policy π , the (unnormalized) state-action occupancy measure is defined as
[Equation]
∀
[Equation]
Given any occupancy measure ν π , the policy π that generates ν π can be recovered as
[Equation]
According to [2], it is well known that the set of all state-action occupancy measures form a polyhedron (cid:8)
[Equation] [Equation]
a ∈A
[Equation]
I − γ P a ) ν a = ρ 0 (cid:9) , where ν a :
= ( ν ( s, a )) s ∈S is an |S| -dimensional column vector, I is the |S|×|S| identity matrix, and P a :
= ( P ( s ′ | s, a ))
s ′ ,s is an |S|×|S| transition matrix, see also [26].
Therefore, combined with the fact that J ( π ) = ⟨ ν π , r ⟩ and J ui ( π ) = ⟨ ν π , u i ⟩ , the CMDP problem (1) can be reformulated as an LP problem with |S| + I constraints:
[Equation]
Due to the fundamental theorem of LP, see e.g. [5], problem (4) has an optimal basic feasible solution with at most |S| + I positive entries, which indicates the following proposition.
Proposition 2.1.
For the CMDP problem (1) with I constraints, there is an optimal policy π ∗ such that | supp( ν π ∗ ) | ≤ N :=
min {|S| + I, |S||A|} , where supp( · ) denotes the support of a vector.
This result captures the potential sparse structure of the optimal policy when I is not as large as |S||A| , and is the key to deriving a tight complexity dependence on the number of constraints I .
2.2 Off-policy learning from demonstration
In this work, we consider the offline CMDP problems where the agent cannot interact with the environment.
Instead, the optimization is conducted using a fixed offline dataset.
To standardize the discussion, we make the following assumption on the offline dataset, see e.g. [21].
Assumption 2.2 (Independent batch dataset) .
The batch dataset D consists of independent tuples ( s, a, s ′ , r, u ) , such that ( s, a ) ∼ µ , E [ r | s, a ] =
r ( s, a ) , E [ u i | s, a ] = u
i ( s, a ) , and s ′ ∼ P ( ·| s, a ) , where µ is called the reference distribution.
To characterize the distribution shift of an arbitrary occupancy measure ν π from the reference distribution µ , we introduce the following notion of the deviation: D π := max s,a (1 − γ )
ν
π ( s,a ) µ ( s,a ) , where the (1 − γ ) -factor normalizes ν π to be a distribution.
In offline RL, it is natural to assume that the deviation D π ∗ of the optimal policy is finite.
That is, the reference distribution µ fully covers supp( π ∗ ) .
Otherwise, no optimality can be guaranteed.
Combining the sparse nature of the optimal solution of (1), we introduce the following finite concentrability assumption for our problem.
Assumption 2.3.
For ∀ ψ ≥ 1 , denote the ψ -deviated policy class as Π( ψ ):= { π : ν π ∈ D ( ψ ) } where
[Equation]
We assume there exists a finite ψ such that some optimal policy π ∗ is contained in Π( ψ ) .
Let C ∗ be the minimum of such ψ .
We call this constant C ∗ the (single-policy) concentrability coefficient.
The above assumption includes a sparsity induced constraint as a result of Proposition 2.1, its counterpart in the definition of single-policy concentrability of offline MDP [21] is the deterministic optimal policy.
The explicit dependence on N in D ( ψ ) facilitates the derivation of the information theoretic lower bound as well as a near-optimal algorithm.
A second remark is that if we know any upper bound ψ of the coefficient C ∗ , then it will be sufficient to only consider the policies in Π( ψ ) .
When C ∗ is unknown, ψ control the risk of distribution shift.
Consequently, in this paper, we propose to solve the LP formulation (4) with a tighter feasible region introduced by D ( ψ ) .
This will allow us to properly control the variance of the off-policy sampling when some of µ ( s, a ) is extremely small or even zero.
We call this strategy deviation control .
2.3 Conservatism toward constraints
We say policy π is safe if it satisfies all constraints in (1), and we say π is ϵ -safe if J ui ( π ) ≥ − ϵ , for ∀ i ∈
[ I ] .
Most of the existing online CMDP algorithms guarantee O (cid:0) 1 / √ T (cid:1) average safeness.
To ensure the true safeness (zero constraint violation) in this work, we assume the Slater’s condition to hold throughout this paper.
In fact, in Section 5, we will show that the Slater’s condition is the necessary condition for any offline CMDP algorithm to obtain zero constraint violation.
Assumption 2.4.
There exists φ > 0 and a policy π such that J ui ( π ) ≥ φ 1 − γ , ∀ i ∈
[ I ] .
A prior knowledge of such a constant φ is assumed throughout our discussion, and we also assume the Slater’s condition holds for Π ′ :
= Π( C ∗ ) .
Given Assumption 2.4, we leverage the idea of conservative constraints proposed in [4].
Namely, instead of J ui ( π ) ≥ 0 , we consider the conservative constraints J u i ( π ) ≥ κ when solving the CMDP problem, where κ > 0 is a properly chosen parameter that controls the level of conservatism in the constraints.
In order to keep the form of the constraints in problem (1), we adopt a shifted utility function u κi defined by u κi ( s, a ) :
= u i ( s, a ) − (1 − γ ) κ for ∀ ( s, a ) ∈ S × A , ∀ i ∈ [ I ] .
Therefore, J ui ( π ) ≥ κ is then equivalent to J u κ i ( π ) ≥ 0 .
It can be shown that a properly selected κ will facilitate a high probability of preserving zero constraint violation, while only introducing an extra O (cid:0) κφ (cid:1) sub-optimality gap in the reward.
The Deviation-controlled Primal Dual Learning (DPDL) algorithm
To solve CMDP with offline samples, we transform its LP formulation (4) to a saddle point form
[Equation]
where D ( ψ ) is defined by (5), V ∈ R |S| , λ ∈ R I are Lagrangian multipliers, and the matrix U κ is defined as U κ :=
[ u κ 1 , · · · , u κI ] ⊤ ∈
R
I ×|S||A| with u κi being the shifted utility defined in Section 2.3.
Given the reference distribution µ , the objective function can be rewritten as an expectation:
L ( V, λ, ν ) =
[Equation]
If the reference distribution µ is known, we can directly sample a stochastic gradient of L .
However, when the reference distribution µ is unknown in practice, then the importance sampling weight ν ( s,a ) µ ( s,a ) is also unknown.
To tackle this issue, let ˆ µ be a proper estimation of the reference distribution µ , we introduce the weights w ( s, a )
= µ ( s,a ) ˆ µ ( s,a ) , and the diagonal matrix W = diag ( w ( s, a )) .
Then we apply a change of variables x
= W − 1 ν , in other words, we set x ( s,a ) ˆ µ ( s,a ) =
ν ( s,a ) µ ( s,a ) for ∀ s, a to enable sampling.
From now on, we will focus on the following reweighted problem
[Equation]
where the feasible regions are defined as
[Equation]
and
[Equation]
The sets X , V and Λ are chosen to be large enough so that they contain the optimal solution of the problem (6), see detailed discussion in Appendix E.
Given a sample ζ = ( s 0 , s, a, s ′ , r, u ) ∼ ρ 0 ×D , and a point Z :=
( V, λ, x ) , we construct the unbiased gradient estimators for L w ( · ) as
[Equation]
where I s is the |S| -dimensional unit vector with the s -th element being one, I s,a is the |S||A| - dimensional unit vector with the ( s, a ) -th
element being one, and u κ = u − κ (1 − γ ) 1 ∈ R
I is the shifted utility vector.
Based on these estimators, we propose a stochastic mirror descent ascent approach to solve problem (7), as stated in Algorithm 1.
The algorithm starts from a feasible solution Z [-
 = = 0 , λ 1 = 1 φI , x 1 =
N |S||A|
ˆ µ 1 − γ .
In each iteration, an offline sample ζ t is used to construct the unbiased gradient estimators g tV , g tλ and g tx .
A stochastic mirror descent ascent step (11) is then used to update the solution Z t , where Proj V ( · ) denotes the Euclidean projection to the set V , and KL( Y ∥ Y ′ ) :
= (cid:80) i
Y i log Y
i Y ′
i − (cid:80) i Y i
+ (cid:80)
i Y
i denotes the generalized KL divergence.
Simple closed form solutions are available to the V t +1 and λ t +1 updates.
By taking the advantage of the special structure of g t x and the fact that x t ∈ X is feasible, the x t +1 subproblem can be reduced to the root finding of a 1-dimensional monotone function, which can be solved efficiently, see details in Appendix A.
Finally, it is worth noting that x is the approximate optimal solution to the reweighted problem.
And Wx will be the approximate solution to the original problem (6) before the change of variable.
Therefore, ideally, we should have output the policy π w ( a | s ) =
[Equation]
) , which is inaccessible in practice without knowing the reference distribution µ .
In order to overcome such dilemma, we show that by properly constructing the estimated distribution ˆ µ , the π output by Algorithm 1 will be close enough to the ideal output π w .
The sample complexity of DPDL 4.1 Main results of DPDL
For the DPDL algorithm, the convergence and performance guarantee of the output policy ¯ π are summarized as the following theorem.
{+input : Tolerance ϵ > 0 , confidential level δ > 0 , conservatism level κ > 0 , stepsize η t > 0 , constants α V , α λ , α x , N e , ς > 0 , and initial feasible solution Z 
+}
=
[ V 1 ; λ 1 ; x 1 ] .
1 Obtain N e samples from D , let N ( s, a ) be the times that the pair ( s, a ) appears.
Compute ˆ µ ( s, a ) = max (cid:18) N
( s, a ) N e , ς (cid:19) , ∀ ( s, a ) ∈ S × A . (10) for t = 1 , · · · , T − 1 do Sample ζ t = ( s 0 t , s t , a t , s ′ t , r t , u t ) from ρ 0 × D ;
Compute stochastic gradients g tV :
= (cid:98)
g V
( Z t ; ζ t ) , g tλ := (cid:98) g λ
( Z t ; ζ t ) , and g tx := (cid:98) g x ( Z t ; ζ t ) ;
Compute the stochastic mirror descent ascent update
[Equation]
11) 5 Compute the average iterate x = 1 T (cid:80) Tt =1 x t , V = 1 T (cid:80)
Tt =1 V t , λ = 1 T (cid:80)
Tt =1 λ t ; 6 Compute π ( a | s ) =
x ( s
a ) (cid:80)
a
x
( s
[Equation]
a ′ )
, for all
( s
a ) ; output: Policy π and the approximate solution x . Theorem 4.1. Suppose that Algorithm 1 runs with η t ≡ 1 √ T , κ = 5 φϵ , α λ = 11 − γ (cid:113) ψ log I , α V = φ (cid:113) ψ |S| , α x = 1 φ (1 − γ ) (cid:113) N ψ log ψ , and ψ ≥ C ∗ .
Then for any fixed ϵ ∈ (cid:0) 0 , 1 10(1 − γ ) (cid:3) , and T ≥ c o
N ψι φ 2 (1 − γ ) 4 ϵ 2 , where ι = log (cid:16) ψ |S||A| I δ (cid:17) and c o is a universal constant, the output policy π of DPDL satisfies the following with probability at least 1 − δ
[Equation]
When ψ = O ( C ∗ ) , DPDL needs at most ˜ O (cid:16) [Equation]
samples to find a safe O ( ϵ ) -optimal policy.
Remark 4.2.
When the prior knowledge of C ∗ is not available, and the selected parameter ψ < C ∗ but the Slater’s condition for Π( ψ ) still holds, the output policy π of DPDL satisfies that
[Equation]
and
[Equation]
where S denotes the set of safe policies, and ϵ approx ( ψ ) :
=
J ( π ∗ ) − max π ∈ Π( ψ ) ∩ S J ( π ) in some sense measures the “sub-optimality” of the policy class Π( ψ ) .
In case a fixed sub-optimality gap ϵ is given, such difficulty of unknown C ∗ also appears in the guarantees provided in previous works [15, 21, 22, 28, 30, 31].
A simple approach to resolve the difficulty of an unknown C ∗ is discussed later in Section 6.
4.2 The analysis of DPDL
We break down the analysis of Theorem 4.1 into the following steps.
First of all, we provide a proper choice of N e and ς
so that ˆ µ is close enough to µ .
See proof in Appendix B.
Proposition 4.3.
Denote ϵ e = ϵ 100 , and let ς = φ (1 − γ ) 2 ϵ e 2 N ψ , and N e ≥
[Equation]
Then with probability at least 1 − δ / 3 , the estimated reference distribution ˆ µ defined by (10) satisfies the following properties simultaneously: (1).
µ ( s,a ) ˆ µ ( s,a ) ≤ 2 , and ˆ µ ( s, a ) ≥ ς , for all s, a ; (2).
For any π ∈ Π( ψ ) , W − 1 ν π ∈ X ; (3).
For any x ∈ X , ∥
Wx − x ∥ 1 ≤ φ (1 − γ )
ϵ e .
All the rest of our analyses are all conditioning on the success of Proposition 4.3.
It is worth noting that in Proposition 4.3, (3) clarifies the validity of constructing the output policy π with x instead of
Wx ; (2) explains why the feasible region X is defined as (8); and (1), combined with the carefully specified feasible domains, provides the proper upper bounds on the magnitude and variance of the unbiased gradient estimators in (9).
A very detailed discussion is provided in Appendix C. In particular, for the (cid:98) g x ( · ) estimator, an explicit O ( N ) dependence has been established for both the magnitude and variance, which plays a crucial role in deriving the optimal O (min {|S||A| , |S| + I } ) dependence on |S| , |A| and I .
Let us define the following gap to measure the performance of the output ¯ x w.r.t.
problem (7):
[Equation]
Based on the properly bounded gradient estimators, a high probability bound for Gap( x ) is established in the following theorem.
Its proof is detailed in Appendix D.
Theorem 4.4.
Suppose the constants η t , α V , α λ , α x and κ are chosen the same as Theorem 4.1.
Then there is a universal constant c o such that, as long as T ≥ c o
[Equation] 2 , the output x satisfies
Gap( x ) ≤ ϵ 2 with probability at least 1 − δ / 3 .
Given Theorem 4.4, we finalize the proof of Theorem 4.1 by properly transforming the bound on
Gap( x ) to the expected reward gap and the constraint violation on the original CMDP problem (1), which is discussed in details in Appendix E.
4.3 Extension to asynchronous setting
In some situations, an independent dataset that satisfies Assumption 2.2 may not be available.
Instead, the dataset may have the following asynchronous structure.
Assumption 4.5.
The asynchronous dataset D async is a single sample trajectory generated by some behavior policy π b .
Namely, what we observe is a sequence { s t , a t , r t , u t } t ≥ 1 generated under π b .
We assume the Markov Chain { ( s t , a t ) } t ≥ 1 is irreducible, aperiodic and uniformly ergodic, with the stationary distribution µ and the mixing time t mix < + ∞ .
The asynchronous data structure introduced here is frequently considered in RL, for example, the asynchronous Q-learning [14].
However, to our best knowledge, this type of offline data has yet been considered under the assumption of a finite single-policy concentrability.
In this situation, we set ζ t =
( s 0 t , s t , a t , s t +1 , r t , u t ) in the DPDL method (Algorithm 1), where s 0 t ∼ ρ
and ( s t , a t , s t +1 , r t , u t ) is the tuple in the t -th time step of the asynchronous dataset.
The sample complexity of the DPDL Algorithm under Assumption 4.5 is established as follows.
Theorem 4.6.
Under Assumption 4.5, we follow the choice of constants in Theorem 4.1.
Then given any fixed ϵ ∈ (cid:16) 0 ,
[Equation]
and T [Equation] 2 , the output policy π of DPDL satisfies the following with probability at least 1 − δ
[Equation]
ϵ and
J [Equation] , ∀ i ∈
[ I ] .
Here ι = log ( T |S||A|
I/δ ) and c ′ o is a universal constant.
Therefore, when ψ = O ( C ∗ ) , DPDL needs at most ˜ O (cid:16) t 2mix N C ∗ φ 2 (1 − γ ) 4 ϵ 2 (cid:17) samples to find a safe ϵ -optimal policy.
The main framework for proving Theorem 4.6 is similar to that in Section 4.2, thus we present the proof in the Appendix H. However, compared to the synchronous setting, a key difficulty here is that the gradient estimators (cid:98) g V ( Z t ; ζ t ) , (cid:98) g
λ ( Z t ; ζ t ) , and (cid:98) g x
( Z t ; ζ t ) are no longer unbiased, because the samples { ζ t } Tt =1 are obtained from a sample path.
This brings further difficulties in the analysis because the variance of the estimators can be amplified by the correlation between samples.
The basic idea to deal with this difficulty is to leverage the mixing property of the uniformly ergodic
Markov chain.
Take the (cid:98) g x ( · ) estimator for example, the bias can be well controlled as long as
T is selected larger than the mixing time t mix of the sample path, which can be illustrated by the following decomposition
[Equation]
[Equation] [Equation] When t = ˜Ω ( t mix ) , one can bound the bias of (cid:98) g x ( Z t ; ζ t ) by ˜ O
( t mix η ) with suitably chosen τ .
Lower Bound of Sample Complexity for Learning CMDP
In this section we will discuss whether the DPDL Algorithm is the near-optimal and whether
the
Slater’s condition (Assumption 2.4) is necessary in achieving zero constraint violation.
We answer these questions affirmatively by establishing the following theorems.
Theorem 5.1.
Suppose S ≥ 4 , A ≥ 3 , I ≥ 8 , C ≥ 2 , γ ∈
[ 12 , 1) , N ≥ 1 .
For any learning algorithm A , there exists a CMDP M =
( S , A , P , r, ( u i )
i ∈
[ I ] , γ, ρ 0 ) and a reference distribution µ , such that the following hold true. [Equation]
the concentrability coefficient [Equation] and µ satisfies [Equation]
(2) Let ˆ π be the policy output by A given N offline samples from µ , and let π ∗ be the optimal policy, then at least one of the following two inequalities hold true:
[Equation]
and E M , A (cid:2) violation(ˆ π ) (cid:3) ≳ 1 , where violation(ˆ π ) :
= (cid:80) Ii =
[ J ui (ˆ π )] − , and J ui is the utility w.r.t.
the constraints J ui ≥ 0 , ∀ i ∈
[ I ] .
For DPDL, the constraint violation is guaranteed to be zero with high probability, then only the first inequality is valid for our method, which indicates an Ω (cid:0) N C ∗ (1 − γ ) 3 ϵ 2 (cid:1) sample complexity lower bound.
Therefore, the complexity of DPDL is nearly optimal up to an ˜ O (cid:0) 11 − γ (cid:1) factor.
Besides the lower bound, we also establish the necessity of the Slater’s condition in ensuring zero violation.
Theorem 5.2.
Let S, A, C , γ be the same as Theorem 5.1.
For any algorithm A , there exists a CMDP
M = ( S , A , P , r, ( u i ) i ∈
[ I ] , γ, ρ 0 ) with I = 1 , |S| ≤ S , |A| ≤ A and a reference distribution µ with
C ∗ ≤ C , such that E M , A [violation(ˆ π )] ≳ min (cid:110) 11 − γ , (cid:113)
SC (1 − γ ) 3 N (cid:111) , where ˆ π is the output policy of
A given N samples from µ .
Theorem 5.2 is obtained by utilizing the same idea as Theorem 5.1.
Thus we only discuss the derivation of Theorem 5.1, while moving all the details to Appendix F.
For offline CMDPs, the fixed data distribution µ fully dominates the frequency of exploring the state-action pairs.
Therefore, intuitively, the hard CMDP instances will be the ones with a large support supp( ν π ∗ ) that widely spreads across the less frequently visited station-action pairs of µ .
Based on this intuition, we design a basic block of CMDP presented in Fig.
1, which is essentially a constrained bandit with 2 K +1 arms.
The instance M will be S replicas of the basic blocks, plus an extra “null” state s − 1 to control C ∗ .
In this discussion, we only consider the case where I ≃ KS , the more general construction that cover full range of I is presented in the appendix. s j 1 r =[-r = −r = r =1 s j ⊕ s j ⊖ s j1 − p q 1 − q p q 1 − q 
-]
r = −r = r =1 s j ⊕ s j ⊖ s j1 − p q 1 − q p q 1 − q 1+ ϖθ i,j1 − ϖθ i,j(a) ( s j 1 , a i ) s
j 1 r = r = −r = r =
s j ⊕ s j ⊖ s j1 − p q 1 − q p q 1 − q 12 (cid:18) 1 − ϖ 2 (cid:19) 12 (cid:18) 1+ ϖ 2 (cid:19) (b) ( s j 1 , b i ) s j 1 r
= r = −r = r =
s j ⊕ s j ⊖ s j1 − p q 1 − q p q 1 − q  (c) ( s j 1 , e )
State, action and transition .
At the states s j ⊕ , s j ⊖ , s j 0 , there is no action to be taken.
At each state s j 1 , there are 2 K + 1 actions a 1 , b 1 , · · · , a K , b K , e .
The transition dynamics of the j th replica under different actions are illustrated in Fig.
1 where the directed arcs and the numbers associated with them are the transitions and the corresponding probabilities, where p = 12 − γ and q = 2 − 1 γ are some constants, while ϖ and θ i,j ∈ {− 1 , 1 } , ∀ i, j are parameters to be designed.
Constraints and Reward .
By carefully selecting the u i ’s, one can construct a set of I = SK constraints that indicate π ( a i | s j 1 ) ≤ π ( b i | s j 1 ) ≤ 14 K , ∀ i, j .
For the reward, we set r ( s j 1 ) =
r ( s j 0 ) = 0 , r ( s j ⊕ )
= 1 , and r ( s j ⊖ ) = − 1 , regardless of the actions.
At any replica j , we can view a i , b i , and e as bandit arms with (cumulative) reward cϖθ i,j , − cϖ 2 , and 0 respectively, for some c > 0 .
When θ i,j = − 1 , one would rather pick e .
But when θ i,j = 1 , due to the constraint π ( a i | s j 1 ) ≤ π ( b i | s j 1 ) ≤ 14 K , picking a i and b i with equal probability 14 K will be optimal.
In fact, this 14 K upper bound forces the support of the optimal policy to widely spread across the ( i, j ) ’s where θ i,j = 1 , and the task of learning is essentially determining whether θ i,j = 1 for each ( i, j ) .
Optimal policy .
Based on the above discussion, it is not hard to see that the unique optimal policy is π [Equation]
[Equation] and π
[Equation]
[-=
 I { θ i,j =1 } .-][Equation]
Finally, with the above π ∗ ,θ and a proper initial distribution ρ 0 , the occupancy measure can be explicitly computed and a reference distribution µ with concentrability coefficient C ∗ ≤ C can be designed.
Moreover, for any policy ˆ π , we consider ˆ θ i,j (ˆ π ):=8 K ˆ π ( a i | s j 1 ) − 1 , then
[Equation]
Namely, if ˆ θ (ˆ π ) is not close enough to the underlying parameter θ , the policy ˆ π will incur a considerable reward gap or constraint violation.
By setting ϖ = min (cid:110)(cid:113) ( SK − 3) C 16(1 − γ ) N , 12 (cid:111) to be a small enough number, any two CMDP instances with different θ parameters will be non-distinguishable, given N samples from µ .
According to [9] and [24], there exists a subset Θ ⊆ {− 1 , 1 } SK such that | Θ | ≥ exp( SK/ 8) , and ∥ θ − θ ′ ∥ 1 ≥ SK 2 for any pair of different θ, θ ′ ∈ Θ .
In other words, there will be at least exp( SK/ 8) CMDP instances with different enough θ parameters while being non-distinguishable under N samples.
Then the rest of the arguments will follow by applying the generalized Fano’s inequality [3].
A detailed proof is provided in Appendix F.
Adaptive deviation-control framework of DPDL
We should notice that in both Theorems 4.1 and 4.6, it has been explicitly emphasized that a prior belief ψ ≥ C ∗ is required.
Otherwise, both the reward and the constraints will suffer an extra loss of ϵ approx ( ψ ) .
In this section, we propose an adaptive deviation-control framework (Algorithm 2) to handle the practical situation where no such prior knowledge is available.
Algorithm input : -optimality ϵ , confidence level δ .
x ( K ) ; ϵ, δ (cid:1) =
=
T RUE then Compute (cid:98) J ( π ( K ) ) as an O ( ϵ ) -accurate estimator of J ( π ( K ) ) , set J K
= (cid:98) J ( π ( K ) ) ; if −∞ < J K ≤ J K − 1 + O ( ϵ ) then Terminate ;
{+Set ψ K +
+}
= 2 ψ K ; output: Policy π ( K ) .
At a high-level, Algorithm 2 consists of the following steps.
Verification For the output x of the DPDL, we develop a verification method V ERIFY ( x ; ϵ, δ ) that, with probability at least 1 − δ , returns T RUE only when following two statements hold: (1).
The vector ν :=
Wx satisfies ∥ (cid:80) a ( I − γ P a ) ν a − ρ
∥ 1 = O ( ϵ ) , which essentially checks whether ν is approximately a valid occupancy measure; (2).
The policy π induced by x is safe.
At step K , if any one of the two statements does not hold, we immediately know ψ K < C ∗ due to the analysis of
Theorem 4.1.
Consequently, we to double the coefficient ψ K +1 ← 2 ψ K in the next iteration.
Certifying performance improvement When V ERIFY ( x ; ϵ, δ ) returns T RUE , then it holds that j 0 ( ψ ) =
J ( π ( K ) )
+ O ( ϵ ) , where j 0 ( ψ ) denotes the optimal value of problem (7) with κ = 0 .
That is, one can estimate j 0 ( ψ ) with (cid:98) J ( π ( K ) ) if V ERIFY ( x ; ϵ, δ )
=
T RUE .
As long as V ERIFY returns T RUE for two consecutive runs, and the performance improvement is small, i.e., j 0 ( ψ K ) − j 0 ( ψ K − 1 ) =
O ( ϵ ) , then Lemma 6.1 guarantees that the safe policy π ( K ) is O ( C ∗ ψ K ϵ )
-optimal.
Lemma 6.1.
The function j 0 ( · ) is strictly increasing in the range ψ ∈ [1 , C ∗ ] , and j 0 ( ψ ) =
J ( π ∗ ) for ψ ≥ C ∗ .
For any ψ < ψ ′ ≤ C ∗ , it holds that
[Equation]
Detailed descriptions of V ERIFY and Adaptive-DPDL are presented in Appendix G, and so does the proof of the following theorem.
Theorem 6.2.
Fixed ϵ ∈ (cid:0) 0 , 1 10(1 − γ ) (cid:3) , δ ∈ (0 , 1) .
Then with probability at least 1 − δ , AdaptiveDPDL stops at step K such that ψ K ≤ 4 C ∗ and outputs the safe policy π ( K ) with sub-optimality gap J ( π ∗ ) − J ( π ( K ) ) ≤ O (cid:16) C ∗ ψ K ϵ (cid:17) .
Moreover, there exists a (problem dependent) constant ϵ 0 ( M ) such that, if ϵ ≤ ϵ 0 ( M ) , then it must hold that ψ K ∈ [ C ∗ , 2 C ∗ ) and π ( K ) is ϵ -optimal.
Intuitively, the Adaptive-DPDL will quickly terminate within O (log 2 C ∗ ) calls of DPDL, resulting in a total samples complexity of ˜ O (cid:16) N C ∗ (1 − γ ) 4 ϵ 2 (cid:17) .
Acknowledgement
J. Zhang is supported by Ministry of Education, Singapore, grant R-266-000-158-133.
Z. Wen is supported in part by the NSFC grant 11831002.
F. Chen is supported in part by the elite undergraduate training program of School of Mathematical Sciences in Peking University.