S AFE E XPLORATION I NCURS N EARLY N O A DDITIONAL S AMPLE C OMPLEXITY FOR R EWARD - FREE RL
A BSTRACT
Reward-free reinforcement learning (RF-RL), a recently introduced RL paradigm, relies on random action-taking to explore the unknown environment without any reward feedback information.
While the primary goal of the exploration phase in RF-RL is to reduce the uncertainty in the estimated model with minimum number of trajectories, in practice, the agent often needs to abide by certain safety constraint at the same time.
It remains unclear how such safe exploration requirement would affect the corresponding sample complexity in order to achieve the desired optimality of the obtained policy in planning.
In this work, we make a first attempt to answer this question.
In particular, we consider the scenario where a safe baseline policy is known beforehand, and propose a unified Safe reWard-frEe
ExploraTion (SWEET) framework.
We then particularize the SWEET framework to the tabular and the low-rank MDP settings, and develop algorithms coined
Tabular-SWEET and Low-rank-SWEET, respectively.
Both algorithms leverage the concavity and continuity of the newly introduced truncated value functions, and are guaranteed to achieve zero constraint violation during exploration with high probability.
Furthermore, both algorithms can provably find a near-optimal policy subject to any constraint in the planning phase.
Remarkably, the sample complexities under both algorithms match or even outperform the state of the art in their constraint-free counterparts up to some constant factors, proving that safety constraint hardly increases the sample complexity for RF-RL.
I NTRODUCTION
Reward-free reinforcement learning (RF-RL) is an RL paradigm under which a learning agent first explores an unknown environment without any reward signal in the exploration phase, and then utilizes the gathered information to obtain a near-optimal policy for any reward function during the planning phase.
Since formally introduced in Jin et al. (2020b), RF-RL has attracted increased attention in the research community (Kaufmann et al., 2021;
Zhang et al., 2020; 2021; Wang et al., 2020; Modi et al., 2021).
It is particularly attractive for applications where many reward functions may be of interest, such as multi-objective RL (Miryoosefi & Jin, 2021), or the reward function is not specified by the environment but handcrafted in order to incentivize some desired behavior of the RL agent (Jin et al., 2020b).
The ability of RF-RL to identify a near-optimal policy in response to an arbitrary reward function relies on the fact that the agent is allowed to explore any action during exploration.
However, in practice, unrestricted exploration is often unrealistic or even harmful.
In order to build safe, responsible and reliable artificial intelligence (AI), the RL agent often has to abide by certain application-dependent constraints, even during the exploration phase.
Two motivating applications are provided as follows
• Autonomous driving.
In order to learn a near-optimal driving strategy, an RL agent needs to try various actions at different states through exploration.
While RF-RL is an appealing approach as the reward function is difficult to specify, it is of critical importance for the RL agent to take safe actions (even during exploration) in order to avoid catastrophic consequences.
• Cellular network optimization.
The operation of cellular network needs to take a diverse corpus of key performance indicators into consideration, which makes RF-RL a plausible solution.
Meanwhile, the exploration also needs to meet certain system requirements, such as power consumption.
While meeting these constraints throughout the learning process is a pressing need for the broad adoption of RL in real-world applications, it is a mission impossible to accomplish if no other information is provided, as the learner has little knowledge of the underlying MDP at the beginning of the learning process and will inevitably take undesirable actions (in hindsight) and violate the constraints.
On the other hand, in various engineering applications, there often exist either rule-based (e.g., autonomous driving) or human expert-guided (e g., cellular network optimization) solutions to ensure safe operation of the system.
One natural question is, is it possible to leverage such existing safe solutions to ensure safety throughout the learning process?
If so, how would the safe exploration requirement affect the corresponding RF-RL performances in terms of the sample complexity of exploration and the optimality and safety guarantees of the obtained policy in planning?
To answer these questions, in this work, we introduce a new safe RF-RL framework.
In the proposed safe RF-RL framework, the agent does not receive any reward information in the exploration phase, but is aware of a cost function associated with actions at a given state.
We require that the cumulative cost in each episode is below a given threshold during exploration, with the aid of a pre-existing safe baseline policy π 0 .
The ultimate learning goal of safe RF-RL is to find a safe and near-optimal policy for any given reward and cost functions after exploration.
Main contributions.
We summarize our main contributions as follows
• First, we introduce a novel safe RF-RL framework that imposes safety constraints during both exploration and planning of RF-RL, which may have implications in various applications.
Second, we propose a unified safe exploration strategy coined SWEET to leverage the prior knowledge of a safe baseline policy π 0 .
SWEET admits general model estimation and safe exploration policy construction modules, thus can accommodate various MDP structures and different algorithmic designs.
Under the assumption that the approximation error function is concave and continuous in the policy space, SWEET is guaranteed to achieve zero constraint violation during exploration, and output a near-optimal safe policy for any given reward function and safety constraint under some assumptions in planning, both with high probability.
Third, in order to facilitate the specific design of the approximation error function to ensure its concavity, we introduce a novel definition of truncated value functions.
It relies on a new clipping method to avoid underestimation of the approximation error captured by the corresponding value function, and ensures the concavity of the resulted value function.
Finally, we particularize the SWEET algorithm for both tabular and low-rank MDPs, and propose Tabular-SWEET and Low-rank-SWEET, respectively.
Both algorithms inherit the optimality guarantee during planning, and the safety guarantees in both exploration and planning.
Remarkably, the sample complexities under both algorithms match or even outperform the state of the art of their constraint-free counterparts up to some constant factors, proving that safety constraint incurs nearly no additional sample complexity for RF-RL.
P RELIMINARIES AND PROBLEM FORMULATION 2. E PISODIC M ARKOV D ECISION P ROCESSES
We consider episodic Markov decision processes (MDPs) in the form of M = ( S , A , P, H, s 1 ) , where S is the state space and A is the finite action space, H is the number of time steps in each episode, P = { P h } Hh =1 is a collection of transition kernels, and P h
( s h
+
| s h , a h ) denotes the transition probability from the state-action pair ( s h , a h ) at step h to state s h +1 in the next step.
Without loss of generality, we assume that in each episode of the MDP, the initial state is fixed at s 1 .
In addition, an MDP may be equipped with certain specified utility functions u
= { u h } Hh =1 , where we assume u h : S ×
A → [0 , 1] is a deterministic function for ease of exposition.
A Markov policy π is a set of mappings { π h : S → ∆( A ) } Hh =1 , where ∆( A ) is the set of all possible distributions over the action space A .
In particular, π h ( a | s ) denotes the probability of selecting action a in state s at time step h .
We denote the set of all Markov policies by X .
For an agent adopting policy π in an MDP M , at each step h ∈
[ H ] where [ H ] := { 1 , . . .
, H } , she observes state s h ∈ S , and takes an action a h ∈ A according to π , after which the environment transits to the next state s h +  with probability P h
( s h
+
| s h , a h ) .
The episode ends after H steps, and we use a virtual state s H +1 to denote the terminal state at step H + 1 .
We use E P,π to denote the expectation of the distribution induced by the transition kernel P and policy π .
Let Q πh,P,u
( s h , a h ) and V πh,P,u ( s h ) be the corresponding action-value function and value function at step h , respectively, for a given collection of utility functions u .
Then, V πh,P,u ( s h ) :=
E P,π (cid:2) (cid:80) Hh ′ =
h u h ′ ( s h ′ , a h ′ )
(cid:12)(cid:12) s h (cid:3) , and Q πh,P,
u ( s h , a h ) :
= E P,π (cid:2) (cid:80) Hh ′ =
h u h ′ ( s h ′ , a h ′ )
(cid:12)(cid:12) s h , a h (cid:3) .
We also use the shorthand V πP,u to denote V π 1 ,P,u ( s 1 ) due to the fixed initial state, and P h
f ( s h , a h ) =
E s h +1 ∼ P h ( ·| s h ,a h ) (cid:2)
f ( s h +1 ) (cid:3) for any function f : S → R .
We further assume that the utility functions are normalized such that for any trajectory generated under a policy, the cumulative value over one episode is bounded by 1, i.e., (cid:80) Hh =1 u h
( s h , a h ) ≤ 1 .
2. S AFE R EWARD -F REE R EINFORCEMENT L EARNING
The safe policy considered in this work is formally defined as follows.
Definition 1.
Given an MDP M ∗ =
( S , A , P ∗ , H, s 1 ) , a set of cost functions
c = { c h } Hh =1 and τ ∈ (0 , 1] , a policy π is ( c, τ ) -safe if V πP ∗ ,c ≤ τ .
Based on this definition of ( c, τ ) -safe policies, we now elaborate the proposed safe RF-RL framework, which contains two phases.
In the first phase of “ exploration ”, the agent is required to efficiently explore the unknown environment without reward signals, and simultaneously not to violate a predefined safety constraint ( c, τ ) in each episode during this exploration phase.
Let π ( n ) be the policy implemented in the n -th episode of the exploration.
Then the agent’s exploration should satisfy the safety constraint in every episode with high probability, namely,
[Equation]
where δ ∈ (0 , 1) and N is the total number of episodes in the exploration phase.
Note that the agent is only given a set of cost functions c but not the reward r in this phase.
This is reasonable for many
RL applications, where the purpose of exploration is not to maximize certain reward but to learn the environment, while the safety constraint need to be satisfied throughout the learning process.
In the second phase of “ planning ”, the agent is given an arbitrary set of reward functions r ∗ and a new set of safety constraint ( c ∗ , τ ∗ ) .
Without further exploration, she is required to learn an ϵ -optimal policy ¯ π with respect to the given reward r ∗ , and subject to the safety constraint ( c ∗ , τ ∗ ) .
Definition 2.
Given an MDP M ∗ =
( S , A , P ∗ , H, s 1 ) , reward functions r ∗ , cost functions c ∗ and τ ∗ ∈ (0 , 1] , ¯ π is an ϵ -optimal ( c ∗ , τ ∗ )-safe policy if
[Equation]
where ϵ ∈ (0 , 1) , and π ∗ is the policy satisfying π
[Equation]
The design goal of safe RF-RL algorithms is three-fold : 1) to collect as few sample trajectories as possible, 2) to satisfy the safety constraint ( c, τ ) in the exploration phase, and 3) to obtain an ϵ -optimal ( c ∗ , τ ∗ )-safe policy for any given reward r ∗ and constraint ( c ∗ , τ ∗ ) in the planning phase.
We note that it is impossible to ensure zero constraint violation with high probability during exploration if an agent starts with no information about the system.
Therefore, we make the assumption that a safe baseline policy is available to the learning agent during exploration.
Besides, we also assume the constrained MDP always has enough feasible solutions, either during exploration or planning.
Assumption 1 (Feasibility) .
The agent has knowledge of a baseline policy π 0 and κ ∈ (0 , τ ) such that
V π 0 P ∗ ,c ≤ τ − κ .
Besides, for any given constraint ( c, τ ) in exploration or planning phases, the safety margin, defined as ∆( c, τ ) :
= τ − min π V
πP ∗ ,c , is bounded away from zero, i.e. ∆( c, τ ) ≥ ∆ min > 0 .
We remark that assuming the existence of a safe baseline policy is reasonable in practice.
Many engineering applications already have existing solutions deployed and verified to be safe, although their reward performances may not necessarily be near-optimal.
Such solutions can naturally serve as the baseline for safe RF-RL.
Additionally, there are practical ways to construct safe baseline policies, e.g., via imitation learning using expert demonstrations, or via policy gradient algorithms to reduce the cost value function to be below the required safety threshold.
This assumption is also widely adopted in the safe RL literature (see Section 6 for more discussions).
T HE SWEET FRAMEWORK
Compared with constraint-free RF-RL, the additional safety requirements during both exploration and planning bring two main challenges in the design of safe RF-RL algorithms.
First, in order to obtain an ϵ -optimal policy for any given reward during planning, it requires all actions to be sufficiently covered in the exploration phase.
In particular, uniform action selection is one of the enablers for reward-free exploration when the state space is undesirably large (Agarwal et al., 2020; Uehara et al., 2021; Modi et al., 2021).
On the other hand, the predefined safety constraint ( c, τ ) may preclude the agent from taking certain actions in exploration, which may affect the estimation accuracy of the environment and degrade the optimality of the output policy in planning.
This dilemma requires a novel design to balance safety and state-action space coverage during exploration.
Second, there may exist safety constraint mismatch between exploration and planning.
Intuitively, the information obtained under a given set of constraint ( c, τ ) during exploration may not provide enough coverage for the optimal policy under another set of constraint ( c ∗ , τ ∗ ) during planning.
How to design the safe exploration algorithm to handle such constraint mismatch is non-trivial.
In this section, we introduce a unified framework for safe reward-free exploration, termed as SWEET.
We will show that the general framework achieves the second and third design objectives, i.e., safe exploration, and ϵ -optimality and ( c ∗ , τ ∗ )-safety for the output policy in planning.
The first design objective, i.e., low sample complexity for exploration, is dependent on the underlying MDP structure and will be investigated in Section 4 and Section 5 for tabular MDPs and low-rank MDPs, respectively.
3. A LGORITHM DESIGN
The SWEET framework relies on several key design components, namely, the ( ϵ 0 , t ) -greedy policy , the approximation error function , and the empirical safe policy set , as elaborated below.
Definition 3 ( ( ϵ 0 , t ) -greedy policy) .
Given ϵ 0 ∈ (0 , 1) and t ∈ { 0 , 1 , · · · , H } , π ′ is an ( ϵ 0 , t ) -greedy version of π if there exists H ⊂
[ H ] with |H| =
t such that π ′ h
= π h for all h / ∈ H , and
[Equation]
Essentially, under an ( ϵ 0 , t ) -greedy version of a given policy π , the agent follows policy π except for t out of H steps, at which with probability ϵ 0 , she takes actions uniformly at random from the state space A .
One critical property of the ( ϵ 0 , t ) -greedy policy is that, the difference between the value functions under the ( ϵ 0 , t ) -greedy policy and its original policy is bounded by ϵ 0 t for any normalized utility function (See Lemma 1 in Appendix A).
The approximation error function U ( ˆ P, π ) measures the uncertainty in the model estimate ˆ P under a policy π .
Specifically, for a given MDP M ∗ , U ( ˆ P, π ) upper bounds the value function difference under ˆ P and P ∗ , i.e. U ( ˆ P, π ) ≥ max u | V π ˆ P,u − V πP ∗ ,u | , where u is a normalized utility function.
The empirical safe policy set , which is critical for constructing safe exploration policies, is defined as
[Equation]
where ˜ κ, ϵ 0 and t are constants satisfying the condition that τ − ϵ 0 t − ˜ κ > τ − κ .
The intuition behind the construction of the empirical safe policy can be explained as follows (Liu et al., 2021): if V π 0 ˆ P,c
+ U ( ˆ P, π 0 ) ≥ τ − ϵ 0 t − ˜ κ , it indicates that ˆ P is not sufficiently accurate.
Thus, the empirical safe policy set only contains the safe baseline policy π 0 .
On the other hand, if
V π 0 ˆ P,c
+ U ( ˆ P, π 0 ) <
τ − ϵ 0 t − ˜ κ , which happens when U ( ˆ P, π 0 ) is sufficiently small, it indicates that ˆ P is sufficiently accurate on π 0 .
Then, we relax the constraint on V
π ˆ P,c
+ U ( ˆ P, π ) from τ − ϵ 0 t − ˜ κ to τ − ϵ 0 t to include π 0 and other policies in the empirical safe policy set.
Since V π ˆ P,c
+ U ( ˆ P, π ) is an upper bound of the true value V πP ∗ ,c for any π , it ensures that V πP ∗ ,c < τ − ϵ 0 t for all π included in C ˆ P, U .
Moreover, all ( ϵ 0 , t ) -greedy versions of such policies satisfy the safety constraint ( c, τ ) .
With those salient components, SWEET proceeds as follows.
At the beginning of each episode, the agent executes a set of behavior policies , which are ( ϵ 0 , t ) -greedy versions of a reference policy π r obtained in the previous episode.
For the first episode, the reference policy would be π 0 .
The general construction of π r will be elaborated below.
By collecting trajectories generated under the behavior policies, the agent updates the estimated model ˆ P and the corresponding approximation error function U ( ˆ P, · ) .
The agent then seeks a reference policy π r that maximizes the approximation error
U ( ˆ P, π ) within the constructed empirical safe policy set.
Intuitively, U ( ˆ P, π ) is an upper bound of certain distance of distributions over trajectories induced by π under ˆ P and P ∗ .
Therefore, π r induces a distribution that captures the most uncertainty in ˆ P .
Choosing π r thus reduces the uncertainty in ˆ P in a greedy fashion.
If U ( ˆ P, π r ) is less than a termination threshold T defined in SWEET, it indicates that the estimated model
ˆ P is sufficiently accurate for the planning task.
The exploration phase then terminates.
Otherwise, the agent continues to the next episode with the new π r .
After termination, SWEET enters the planning phase and receives arbitrary reward functions r ∗ and a safety constraint ( c ∗ , τ ∗ ) .
The agent utilizes ˆ P to compute a policy ¯ π , which maximizes V π ˆ P,r ∗ subject to an empirical safety constraint
V π ˆ P,c
+ U ( ˆ P, π ) ≤ τ ∗ .
Algorithm 1 has the detail of SWEET.
1: Input
2: // Exploration: 3: while TRUE do 4:
Construct a set of ( ϵ 0 , t ) -greedy policies of π r (see Definition 3) and use them to collect data; 5:
Model estimation: Update ˆ P using collected data; 6: Obtain π r = arg max π ∈C ˆ P, U (˜ κ,ϵ 0 ,t ) U ( ˆ P, π ) where C ˆ P, U (˜ κ, ϵ 0 , t ) is defined in Equation (3); 7: if V π
[Equation]
P,c
+ U ( ˆ P, π 0 )
T HEORETICAL ANALYSIS
Before we present the theoretical guarantee for SWEET, we first introduce the notion of mixture policies and equivalent policies and characterize the concavity over Markov policy space.
Definition 4.
Given two Markov policies π, π ′ ∈ X , we use γπ ⊕ (1 − γ ) π ′ to denote the mixture policy that uses π with probability γ and uses π ′ with probability 1 − γ during an episode.
Definition 5.
Given an MDP M , two policies, including mixture policies, are equivalent if they induce the same marginal distribution over any state-action pair ( a, s ) in any step
h ∈
[ H ] .
By Theorem 6.1 in Altman (1999), for any mixture policy γπ ⊕ (1 − γ ) π ′ , there exists an equivalent
Markov policy π γ ( π, π ′ )
∈ X .
For ease of presentation, in the following, we simply use π γ to denote it when the definition is clear from the context.
Therefore, the Markov policy space X is equipped with an abstract convexity by mapping all mixture policies to their equivalent Markov policies in X .
With this convexity, we can define concave functions on X as follows.
Definition 6.
A function f : X → [0 , 1] is concave and continuous on the Markov policy space X if for any π, π ′ ∈ X and γ ∈ [0 , 1] , f ( π γ ) ≥ γ f ( π )
+ (1 − γ ) f
( π ′ ) , and is continuous in γ ∈ [0 , 1] .
With Definition 6, we have the following result of SWEET.
Theorem 1 ( ϵ -optimality and safety guarantee of SWEET) .
Given an MDP M ∗ and model estimate ˆ P , assume U ( ˆ P, π ) is concave and continuous over the Markov policy space X and (cid:12)(cid:12)
V πP ∗ ,u − V π ˆ P,u (cid:12)(cid:12) ≤
U ( ˆ P, π ) for any normalized utility u and policy π , and Assumption 1 holds.
Let ϵ 0 , t and ˜ κ be constants that satisfy ϵ 0 t + ˜ κ < κ.
Let U = min (cid:110)
ϵ 2 , ∆ min 2 , ϵ ∆ min 5 , τ − ϵ 0 t 4 , ˜ κ (∆( c,τ ) − ϵ 0 t − ˜ κ ) 4(∆( c,τ ) − ϵ 0 t ) (cid:111) , and
T ≤ (∆( c, τ ) − ϵ 0 t ) U / 2 be the termination condition of SWEET.
If SWEET terminates in finite episodes, then, the following statements hold:
(i)
The exploration phase is safe.
(ii)
The output ¯ π of SWEET in the planning phase is an ϵ -optimal ( c ∗ , τ ∗ ) -safe policy.
The detailed proof of Theorem 1 is deferred to Appendix A.
We highlight the main idea behind the analysis as follows.
While the construction of C ˆ P, U (˜ κ, ϵ 0 , t ) ensures safe exploration, the ability for
SWEET to find an ϵ -optimal ( c ∗ , τ ∗ ) -safe policy in planning relies on the concavity and continuity of U ( ˆ P, · ) .
Note that when SWEET terminates, it is only guaranteed that the approximation error U ( ˆ P, π ) is upper bounded by T for the policies within C ˆ P, U (˜ κ, ϵ 0 , t ) .
Due to a possibly different constraint in planning, it is desirable to have
U ( ˆ P, π ) sufficiently small under any π , so that the agent is able to achieve the learning goal in planning with the estimated model
ˆ P .
Let ˜ π = arg max π U
( ˆ P, π ) , π γ be the equivalent Markov policy of γ ˜ π ⊕ (1 − γ ) π 0 , f ( γ ) =
U ( ˆ P, π γ ) and g ( γ ) =
V π γ ˆ P,c .
Then, f is concave and g is linear in γ by Theorem 6.1 in Altman (1999).
Let F ( γ ) =
f ( γ ) + g ( γ ) .
The definition of C ˆ P, U (˜ κ, ϵ 0 , t ) ensures that F (0) ≤ τ − ϵ 0 t − ˜ κ , and F ( γ ) ≤ τ − ϵ 0 t if π γ lies in C ˆ P, U .
It suffices to consider the case when F (1) > τ − ϵ 0 t , under which we can show that both g and f increase with γ .
Then, the concavity of f and linearity of g ensure that f ( γ ) − [Equation] (0) f (1) − f (0) ≥ F ( γ ) − F (0) F (1) − F (0) , as illustrated in Figure 1.
Let π γ 0 be the policy under which F ( γ 0 )
=
τ − ϵ 0 t .
Then, F ( γ 0 ) − F (0) ≥ ˜ κ .
Combining with the fact that F (1) − F (0) ≤ 2 , we have f (1) ≤ f (0) + ( f ( γ 0 ) − f (0)) 2˜ κ ≤ 2 T / ˜ κ , which provides an upper bound on U ( ˆ P, π ) for any π .
3. T RUNCATED VALUE FUNCTION
Theorem 1 highlights the importance of the concave and continuous approximation error function on the Markov policy space.
In the following, we introduce a prototype function, coined truncated value function , which is concave and continuous on the Markov policy space and can be used for the construction of the approximation error functions for both tabular and low-rank MDPs.
Definition 7 (Truncated value function) .
Given an MDP M , α > 0 , and a set of (un-normalized) utility functions u , the truncated value function ¯ V α, · P,u =
¯ V α, · 1 ,P,u ( s 1 ) :
X → R is defined as follows:
[Equation]
where V α,π H +1 ,P,u ( s H +1 )
=
and we omit the upper index α for simplicity when α = 1 .
It is worth noting that the clipping technique is applied to the value function as opposed to the action-value function, where the latter is more conventional in the existing literature.
This new method is critical for achieving the superior sample complexity in safe RF-RL, as will be elaborated later.
Meanwhile, it preserves the desired concavity and continuity (see Lemma 3 in Appendix A), which ensures the safety guarantee for both exploration and planning.
T HE T ABULAR -SWEET ALGORITHM 4. A LGORITHM DESIGN
Under tabular MDPs, state space S and action space A are both finite (with sizes S and A , respectively).
We instantiate the modules of model estimation and exploration policy construction of the
SWEET framework, and specify the approximation error function and parameter selection as follows.
The details of the Tabular-SWEET algorithm is shown in Algorithm 2 in Appendix B.
Model estimation.
At each episode n , the agent uses π ( n − 1) , which is the reference policy derived from the last episode n − 1 , to collect a trajectory { s ( n ) 1 , a ( n ) 1 , . . .
, s ( n ) H , a ( n ) H } .
For that, we set ϵ 0 and t as 0 , i.e., it is essentially a (0 , 0) -greedy version of policy π ( n − 1) .
We note that although π ( n − 1) is a greedy policy, the uncertainty captured by the approximation error function U ( ˆ P, π ) will guide the agent to explore the uncertain state-action pairs and obtain sufficient coverage for the entire space.
The agent then adds new data triples { s ( n ) h , a ( n ) h , s ( n )
h +1 } Hh =1 to a maintained dataset
D .
Let N ( n ) h ( s h , a h )
=
(cid:80) nm =1 1 { s ( n ) h
= s h , a ( n ) h = a h } and N ( n ) h
( s h , a h , s h +1 )
= (cid:80) nm =1 1 { s ( n ) h
= s h , a ( n ) h
= a h , s ( n ) h +
=
s h +1 } be the visitation counters.
The agent estimates ˆ P ( n ) h ( s h
+
| s h , a h ) as N ( n ) h ( s h ,a h ,s h +1 ) N ( n )
h
( s h ,a h ) if N ( n ) h
( s h , a h ) > 1 and as 1 S otherwise.
Approximation error function.
Inspired by M´enard et al.
(2021), we adopt an uncertainty-driven virtual reward function ˆ b ( n ) h
( s h , a h ) =
β 0 H N ( n ) h ( s h ,a h ) to guide the exploration, where β 0 is a fixed parameter.
Let α H = 1 + 1 /H .
Then, the approximation error function is specified as U ( n ) ( π ) := 4 (cid:113)
¯ V α H ,π ˆ P ( n ) , ˆ b ( n ) .
According to Lemma 3, U ( n ) ( π ) is concave and continuous in π .
Besides, as shown in Lemma 8 in Appendix B, we have | V πP ∗ ,u − V π ˆ P,u | ≤ U ( n ) ( π ) for any normalized utility u , i.e.,
U ( n ) ( π ) is a valid upper bound on the estimate error for the corresponding value function.
The required properties of U in Theorem 1 are thus satisfied.
Exploration policy.
To guarantee that the exploration is safe, we set ˜ κ = κ/ 2 , and construct an empirical safety set C ( n ) :
=
C ˆ P ( n ) , U ( n ) ( κ/ 2 , 0 , 0) (Equation (3)).
Hence, the algorithm finds a policy π ( n ) used for the next episode, which is in the safe set C ( n ) and maximizes the truncated value function
¯ V α H ,π ˆ P ( n ) , ˆ b ( n ) .
The exploration phase stops at episode n ϵ when U ( n ϵ ) ( π ( n ϵ ) ) ≤ T .
The algorithm will utilize the model learned in episode n ϵ to design an ϵ -optimal policy with respect to arbitrary given reward r ∗ and safety constraint ( c ∗ , τ ∗ ) .
4. T HEORETICAL ANALYSIS
The theoretical guarantee of Tabular-SWEET is characterized in the theorem below, whose proof can be found in Appendix B.
Theorem 2 (Sample complexity of Tabular-SWEET) .
Given ϵ, δ ∈ (0 , 1) , and safety constraint ( c, τ ) , under Assumption 1, let U = min (cid:110)
ϵ 2 , ∆ min 2 , ϵ ∆ min 5 , τ 4 , κ 16 (cid:111) , and T = ∆( c, τ ) U / 2 be the termination condition of Tabular-SWEET.
Then, with probability at least 1 − δ , Tabular-SWEET achieves the learning objective of safe reward-free exploration (Equations (1) and (2) ), and the number of trajectories collected in the exploration phase is at most ˜ O (cid:16) HSA ( S +log [Equation] 1 /δ ))
∆( c,τ )
[Equation]
We discuss several possible scenarios and the corresponding selections of U as follows.
• Constraint-free RF-RL.
For this case ∆( c, τ ) =
∆ min = κ = 1 , and c = 0 .
Thus, U =
Θ( ϵ ) and the sample complexity is ˜ O (cid:0) HS
A/ϵ 2 (cid:1) , which matches the state of the art (M ´ enard et al., 2021).
• Constraint-free planning.
If only safe exploration is required, we set U =
Θ(min { ϵ, κ } ) , and the sample complexity scales in ˜ O (cid:16) HS
A ∆( c,τ ) 2 ( 1 ϵ 2 + 1 κ 2 ) (cid:17) .
The blow-up factor 1 ∆( c,τ ) 2 depends on the safety margin, and the impact of baseline policy only appears in the ϵ -independent term.
• Constraint mismatch between exploration and planning.
For this case, we set U =
Θ( ϵ ∆ min ) , and the sample complexity is at most ˜ O (cid:16) HS
A ∆( c,τ ) 2 ( 1 ϵ 2 ∆ min2 + 1 κ 2 ) (cid:17) .
T HE L OW - RANK -SWEET ALGORITHM 5. L OW - RANK MDP
In this section, we present another SWEET variant for low-rank MDPs. Definition 8 (Low-rank MDP (Jiang et al., 2017; Agarwal et al., 2020; Uehara et al., 2021)) .
An MDP M is a low-rank MDP with dimension d ∈ N if for each h ∈
[ H ] , the transition kernel P h admits a d -dimensional decomposition, i.e., there exist two features ϕ h : S ×
A → R d and µ h : S → R d such that P h
( s h
+
| s h , a h ) = ⟨ ϕ h ( s h , a h ) , µ h
( s h +1 ) ⟩ , ∀ s h , s h +1 ∈ S , a h ∈ A .
Let ϕ =
{ ϕ h } h ∈ [ H ] and µ = { µ ∗ h } h ∈
[ H ] be the features for P .
Then, ∥ ϕ ∗ h ( s, a ) ∥ 2 ≤ 1 , ∥ (cid:82) µ ∗ h ( s ) g ( s ) ds ∥ 2 ≤ √ d , ∀ ( s, a ) ∈ S × A , ∀ g : S → [0 , 1] .
Differently from linear MDPs (Wang et al., 2020; Jin et al., 2020b), low-rank MDP does not assume that the features ϕ are known a priori.
The lack of knowledge on features in fact invokes a nonlinear structure, which makes it impossible to learn a model in polynomial time if there is no assumption on features ϕ and µ .
We hence adopt the following conventional assumption (Jiang et al., 2017; Agarwal et al., 2020; Uehara et al., 2021) from the recent studies on low-rank MDPs.
Assumption 2 (Realizability) .
A learning agent can access a finite model class { (Φ , Ψ) } that contains the true model, i.e., ( ϕ ∗ , µ ∗ ) ∈
Φ × Ψ , where ⟨ ϕ ∗ h ( s h , a h ) , µ ∗ ( s h +
⟩ = P ∗ h ( s h
+
| s h , a h ) .
We note that finite model class assumption can be relaxed to the infinite case with bounded statistical complexity (Sun et al., 2019; Agarwal et al., 2020).
Then, we present the following standard oracle as a computational abstraction, which is commonly adopted in the literature (Agarwal et al., 2020; Uehara et al., 2021).
Definition 9 (MLE oracle) .
Given the model class (Φ , Ψ) and a dataset D of ( s h , a h , s h +1 ) , the
MLE oracle MLE( D ) takes D as the input and returns the following estimators as the output:
[Equation]
5. A LGORITHM DESIGN
The instantiated SWEET algorithm, termed as Low-rank-SWEET, can be found in Algorithmin Appendix C. It proceeds as follows.
In each iteration n during the exploration phase, the agent samples H trajectories, indexed by { ( n, h ) } Hh =1 .
During the ( n, h ) -th episode, the agent executes an ( ϵ 0 , 2) -greedy version of the reference policy π ( n − 1) , where ϵ 0 = κ/ 6 and the ϵ 0 -greedy action selection only takes place at time steps h and h − 1 .
Denote the trajectory collected in episode ( n, h ) as { s ( n,h ), a ( n,h ), . . .
, s ( n,h ) H , a ( n,h ) H } .
The agent maintains a dataset D h for each time step h , which is updated through D ( n ) h ← D ( [Equation] [Equation] [Equation]
[Equation]
( n,h ) [Equation] , s ( n,h ) h +1 } . Note that both s ( n,h ) h and a ( n,h ) h are affected by the ϵ 0 -greedy action selection.
Model estimation.
Then, the agent obtains the model estimate ˆ P ( n ) through the MLE oracle: (ˆ ϕ ( n ) h , ˆ µ ( n ) h ) = MLE [Equation] D h ) , and ˆ P ( n )
[Equation]
s h
[Equation]
ˆ
[Equation]
[Equation]
Approximation error function.
The algorithm will also use the estimated representation ˆ ϕ ( n ) h to update the empirical covariance matrix ˆ U ( n ) as
[Equation]
It is worth noting that only a ( m,h +1) h is affected by the ϵ 0 -greedy action selection, which is different from the dataset augmentation step.
Next, the agent uses both ˆ ϕ ( n ) h and ˆ U ( n ) to derive an explorationdriven virtual reward function as ˆ b ( n ) h ( s, a ) =
ˆ α ∥ ˆ ϕ ( n ) h ( s, a ) ∥
(ˆ U ( n ) h ) − 1 where ∥ x ∥ A := √ x ⊤ Ax and ˆ α is a pre-determined parameter.
As shown in Lemma 14 in Appendix C, the approximation error can be bounded by the truncated value function with factor α = 1 up to a constant additive term, i.e. | V πP ∗ ,u − V π [Equation] P,u |
[Equation]
P ( n ) , ˆ
[Equation] ( n [Equation] + (cid:113) ˜
Aζ/n := U ( n ) L ( π ) , where “ L ” stands for “Low-rank”.
Exploration policy.
Based on SWEET, we choose ˜ κ = κ/ 3 such that ˜ κ + ϵ 0 t < κ .
Then, the algorithm defines the empirical safe policy set as C ( n ) L :=
C ˆ P ( n ) , U ( n ) L ( κ/ 3 , κ/ 6 , 2) .
It then finds a reference policy π ( n ) in C ( n ) L that maximizes U πL ( π ) , which is used for exploration at the next iteration.
5. T HEORETICAL ANALYSIS
We summarize the results of Low-rank-SWEET in Theorem 3, and defer the proof to Appendix C.
Theorem 3 (Sample complexity of Low-rank-SWEET) .
Given ϵ, δ ∈ (0 , 1) , and safety constraint ( c, τ ) , let U = min (cid:110) ϵ 2 , ∆ min 2 , ϵ ∆ min 5 , τ 6 , κ 24 (cid:111) , and T = ∆( c, τ ) U / 3 be the termination condition of
Low-rank-SWEET.
Then, under Assumption 1,2, with probability at least 1 − δ , Low-rank-SWEET achieves the learning objective of safe reward-free exploration (Equations (1) and (2) ) and the number of trajectories collected in the exploration phase is at most ˜ O (cid:16) H 3 d 4 A 2 log(1 /δ ) κ 2 ∆( c,τ ) 2 U+ H 3 d 4 A 2 log(1 /δ )
κ(cid:17) .
Remark 1 .
For the constraint-free scenario, we set ∆( c, τ ) =
∆ min = κ = 1 , U =
Θ( ϵ ) , and c to be zero.
Then, the sample complexity scales as ˜ O (cid:0) H
d 4 A 2 /ϵ 2 (cid:1) , which outperforms the best known sample complexity of RF-RL (Agarwal et al., 2020; Modi et al., 2021) and even reward-known RL with computational feasibility (Uehara et al., 2021), all for low-rank MDPs.
R ELATED WORKS
Reward-free reinforcement learning.
Reward-free exploration is formally introduced by Jin et al.
(2020a) for tabular MDP, where an algorithm called RF-RL-Explore is proposed, which achieves ˜ O (cid:0) H 3 S 2 A/ϵ 2 (cid:1) sample complexity 1 .
The result is then improved to ˜ O (cid:0) H 2 S 2 A/ϵ 2 (cid:1) by
Kaufmann et al.
By leveraging an empirical Bernstein inequality, RF-Express (M´enard et al., 2021) achieves ˜ O (cid:0) HS 2 A/ϵ 2 (cid:1) sample complexity, which matches the minimax lower bound in H (Domingues et al., 2020).
Zhang et al.
(2020) considers the stationary case, and achieves ˜
O (cid:0) S 2 A/ϵ 2 (cid:1) sample complexity, which is nearly minimax optimal.
When structured MDPs are considered, Wang et al.
(2020) studies linear MDPs and obtains ˜ O (cid:0) d 3 H 4 /ϵ 2 (cid:1) sample complexity, where d is the dimension of feature vectors.
Zhang et al.
(2021) investigates linear mixture MDPs and achieves ˜ O (cid:0) H 3 d 2 /ϵ 2 (cid:1) sample complexity.
Zanette et al.
(2020b) considers a class of MDPs with low inherent Bellman error introduced by Zanette et al.
(2020a).
Agarwal et al.
(2020) studies low-rank MDPs and proposes FLAMBE, whose learning objective can be translated to a reward-free learning goal with sample complexity
˜ O
(cid:0) H 22 d 7 A 9 /ϵ 10 (cid:1) .
Subsequently, Modi et al.
(2021) proposes a model-free algorithm MOFFLE for low-nonnegative-rank MDPs, for which the sample complexity scales as ˜ O
( H 5 A 5 d 3 LV ϵ 2 η ) , where d LV denotes the non-negative rank of the transition kernel.
Recently,
Chen et al.
(2022) studies RF-RL with more general function approximation, but their result scales in ˜
O ( H 6 d 3 A/ϵ 2 ) when specializes to low-rank MDPs, and cannot recover our upper bound.
Safe reinforcement learning.
Safe RL is often cast in the Constrained MDP (CMDP) framework (Altman, 1999) under which the learning agent must satisfy a set of constraints (Efroni et al., 2020;
Turchetta et al., 2020; Zheng & Ratliff, 2020; Qiu et al., 2020; Ding et al., 2020; Kalagarla et al., 2020;
Liu et al., 2021; Wei et al., 2022; Ghosh et al., 2022).
However, most of the constraints considered in the existing works require the cumulative expected cost over a horizon falling below a certain threshold, which is less stringent than the episodic-wise constraint imposed in this work.
Other forms of constraints such as minimizing the variance (Tamar et al., 2012) or more generally maximizing some utility function (Ding et al., 2021), have also been investigated.
Amani et al.
(2021) studies safe RL with linear function approximation, where the constraint is defined using an (unknown) linear cost function of each state and action pair.
In particular, Miryoosefi & Jin (2021) utilizes a reward-free oracle to solve CMDP which, however, does not have any safety guarantee for the exploration phase.
By assuming availability of a safe baseline policy, Zheng & Ratliff (2020) considers a known MDP with unknown rewards and cost functions and presents C-UCRL that achieves regret ˜ O
( N 3 / 4 ) and zero constraint violation, where N is the number of episodes.
Liu et al.
(2021) improves the result by proposing OptPess-LP, and achieves a regret of ˜ O (cid:16) H 2 √ S 3 AN/κ (cid:17) , where κ is the cost value gap between the baseline policy and the constraint boundary.
Safe baseline policy has been widely utilized in conservative RL as well, which is a special case of safe RL, as the constraint is defined in terms of the total expected reward being above a threshold (Garcelon et al., 2020; Yang et al., 2021).
C ONCLUSION
We proposed a novel safe RF-RL framework where safety constraints are imposed during the exploration and planning phases of RF-RL.
A unified algorithmic framework called SWEET was developed, which leverages an existing baseline policy to guide safe exploration.
Leveraging a concave approximation error function, SWEET can achieve zero constraint violation in exploration and provably produce a near-optimal safe policy for any given reward function and safety constraint he feasible assumption in planning.
We also instantiated SWEET to both tabular and low-rank MDPs, resulting in Tabular-SWEET and Low-rank-SWEET.
The sample complexities of both algorithms match or outperform the state of the art in their constraint-free counterparts, proving that the safety constraint does not fundamentally impact the sample complexity of RF-RL.
A CKNOWLEDGMENTS
The work of R.Huang and J. Yang was supported by the U.S. National Science Foundation under the grant CNS-2003131.
The work of Y. Liang was supported in part by the U
S. National Science
Foundation under the grant RINGS-
Supplementary Material A A NALYSIS OF THE SWEET FRAMEWORK AND THE TRUNCATED VALUE FUNCTION
In this section, we first prove two supporting lemmas in Appendix A.1, which are useful for the proof of Theorem 1.
Then, we provide the proof for Theorem 1 in Appendix A.2 and the proof for an important concavity property of the truncated value function in Appendix A.3, which are essential for instantiating Theorem 1 for Tabular-SWEET and Low-rank-SWEET.
A. S UPPORTING LEMMAS
We first show that the value function under an ( ϵ 0 , t ) -greedy policy deviates from that under the original policy by at most ϵ 0 t .
Lemma 1.
Let π ′ be an ( ϵ 0 , t ) -greedy version of policy π .
Then, for an MDP with transition kernel
P and normalized utility function u , we must have
[Equation]
Proof.
First, we prove the statement for the case when t = 1 .
Assume policy π ′ deviates from policy π at step h , and denote ρ πh ( s h ) as the marginal distribution induced by π under the transition kernel P .
Let π U
= ( π 1 , · · · , π h − 1 , U , π h +1 , · · · , π H ) , where U is the uniform policy over the action space, i.e., U ( a h | s h ) = 1 / |A| .
Consider the equivalent Markov policy of ϵ 0 π U ⊕ (1 − ϵ 0 ) π , denoted by π ϵ 0 .
By Lemma 16, we have
[Equation]
where the second equality follows from the fact that
ρ π U h
( s h )
=
ρ πh ( s h ) , since the first h − 1 policies of π and π U are the same.
For any h ′ ≥ h + 1 , we have
[Equation]
where the last equality is due to the definition of π ′ .
Therefore, π ′ =
π ϵ 0 , which further yields that
[Equation]
Since the value function is upper bounded by 1 with normalized utility function u , we immediately obtain that
[Equation]
For the general case where π ′ differs from π at steps in H ⊂ [ H ] with |H| =
t ≤ H , consider a sequence of subsets {H i } t i =
such that H i ⊂ H i +1 , |H
i +1 | − |H
i | = 1 , and H t
= H .
Then, we can define a sequence of policies { π i } ti =1 , such that π t is the ( ϵ 0 , i ) -greedy policy of π that deviates from π at steps in H i .
Then, by the definition of ( ϵ 0 , t ) -greedy policy in Definition 3, π i +1 is an ( ϵ 0 , 1) -greedy version of policy π i .
Thus, by induction, we conclude that (cid:12)(cid:12)(cid:12)
[Table]
where we denote π ′ as π t and π as π 0 .
The following lemma is critical for handling constraint mismatch not only between the exploration phase and the planning phase, but also between the constraint adopted for the construction of the empirical safe policy set used in exploration and the true constraint V πP ∗ ,c ≤ τ .
Lemma 2.
Consider a set X in which the convex combination is defined through γx ⊕ (1 − γ ) y ∈ X , where x, y ∈ X and γ ∈ [0 , 1] .
Let f, g : X → [0 , 1] be two functions on X such that f is concave
and g is convex, i.e. f ( γx ⊕ (1 − γ ) y ) ≥ γf ( x ) + (1 − γ )
f ( y ) and g ( γx ⊕ (1 − γ ) y ) ≤
γg ( x ) + (1 − γ ) g
( y ) .
We further assume that both f, g are continuous w.r.t.
γ ∈ [0 , 1] .
Define an optimization problem (P) as follows:
[Equation]
Assume there exists a strictly feasible solution x 0 ∈ X such that g ( x 0 ) + f ( x 0 ) ≤ ˜ τ − ˜ κ where ˜ κ ∈ (0 , ˜ τ ) , and denote x ∗ as an optimal solution to (P).
If the optimal value of (P) is strictly less than ˜ κ , i.e. f ( x ∗ ) <
˜ κ , then
[Equation]
Proof.
Let x 1 = arg max x ∈ X f ( x ) , and x γ = γx 1 ⊕ (1 − γ ) x 0 ∈ X , which is the convex combination of x 0 and x 1 .
If x 1 satisfies the constraint in (P), the result is trivial.
Therefore, it suffices to consider the case when g ( x 1 ) + f ( x 1 ) > ˜ τ .
We first show that g ( x 1 ) ≥ g ( x 0 ) through contradiction.
Assume g ( x 1 ) <
g ( x 0 ) .
Then, we have
[Equation]
where ( i ) follows because x 0 is a strictly feasible solution, and ( ii ) follows from the assumption that f ( x 0 ) ∈ [0 , 1] and g ( x 0 )
> g ( x 1 ) .
Note that f ( x 0 ) <
˜ κ , and f ( x γ ) is a continuous function with respect to γ ∈ [0 , 1] .
Thus, we can choose γ 1 ∈ (0 , 1) such that
[Equation]
In addition, by the convexity of g and the assumption that g ( x 1 ) < g ( x 0 ) , we have
[Equation]
Combining Equations (7) and (8), we have
[Equation]
which implies that x γ 1 is a feasible solution of the optimization problem (P).
Thus, by the optimality of x ∗ , we have ˜ κ > f ( x ∗ ) ≥ f
( x γ 1 ) = ˜ κ , which is a contradiction.
Therefore, g ( x 0 ) ≤ g ( x 1 ) .
Then, let γ 0 be the solution to the following equation.
[Equation]
Since g ( x 0 ) + f ( x 0 ) ≤ ˜ τ − ˜ κ and f, g ∈ [0 , 1] , we have:
[Equation]
which implies γ 0 ≥ ˜ κ/ 2 .
Since f is concave and continuous w.r.t γ , there exists γ ∗ ≤ γ 0 such that
[Equation]
On the other hand, due to the convexity of g , we have
[Equation]
) where ( i ) follows from the fact that γ ∗ ≤ γ 0 and g ( x 1 ) > g ( x 0 ) .
Combining Equations (10) and (11), we have
[Equation]
which indicates that x γ ∗ is a feasible solution of the optimization problem (P).
Thus, by the optimality of x ∗ , Equation (10), and γ 0 ≥ ˜ κ/ 2 , we conclude that
[Equation]
A. P ROOF OF T HEOREM
We first formally restate Theorem 1 below and then provide the proof for this theorem.
Theorem 4 (Restatement of Theorem 1) .
Given an MDP M ∗ and model estimate ˆ P , assume U ( ˆ P, π ) is concave and continuous over the Markov policy space X and (cid:12)(cid:12)
V πP ∗ ,u − V π ˆ P,u (cid:12)(cid:12)
≤ U ( ˆ P, π ) for any normalized utility u and policy π .
Let ϵ 0 , t and ˜ κ be constants that satisfy ϵ 0 t
+ ˜ κ < κ.
Let
U = min (cid:110) ϵ 2 , ∆ min 2 , ϵ ∆ min 5 , τ − ϵ 0 t 4 , ˜ κ (∆( c,τ ) − ϵ 0 t − ˜ κ ) 4(∆( c,τ ) − ϵ 0 t ) (cid:111) , and T ≤ (∆( c, τ ) − ϵ 0 t ) U / 2 be the termination condition of SWEET.
If SWEET terminates in finite episodes, then, the following statements hold:
(i)
The exploration phase is safe (See Equation (1) ).
(ii)
The output ¯ π of SWEET in the planning phase is an ϵ -optimal ( c ∗ , τ ∗ ) -safe policy (See Equation (2) ).
Proof.
The proof consists of three steps: Step 1 shows that the exploration phase of SWEET is safe;
Step 2 shows that SWEET can find an ϵ -optimal policy in the planning phase for any given reward and without the constraint requirement; and Step 3 shows that SWEET can find an ϵ -optimal policy in the planning phase for any given reward and under any constraint ( c ∗ , τ ∗ ) requirement.
We next provide details for each step.
Step 1.
This step shows that the exploration phase of SWEET is safe.
Note that the exploration policy, denoted by π b , is an ( ϵ 0 , t ) -greedy version of the reference policy π r , where π r is a solution to the following optimization problem:
[Equation]
where
C
[Equation]
otherwise .
If π r = π 0 , then by Lemma 1, we have
[Equation]
where the last inequality is due the condition ϵ 0 t
+ ˜ κ < κ .
If π r ̸ = π 0 , then V π r ˆ P,c + U ( ˆ P, π r ) ≤ τ − ϵ 0 t.
By Lemma 1, we have
[Equation]
where ( i ) follows from the definition of U ( ˆ P, π ) .
Therefore, the exploration phase is safe.
Step 2: This step shows that SWEET can find an ϵ -optimal policy in the planning phase for any given reward r ∗ in the constraint-free setting ( τ ∗ = ∞ ), i.e., the planning phase does not have a constraint requirement.
Consider the Markov policy space X with the convex combination defined by the mixture policy γπ ⊕ (1 − γ ) π ′ , and g ( π )
=
V π ˆ P,c .
Let π r be the reference policy when the termination condition is satisfied.
Then, by the property of
U ( ˆ P, π ) and the termination condition in SWEET, the following statements hold:
• g ( π ) is convex (linear) and U ( π ) is concave on X .
Moreover, they are both continuous.
• The baseline policy π 0 ∈ X and g ( π 0 )
+ U ( ˆ P, π 0 ) ≤ τ − ϵ
t − ˜ κ .
• π r = arg
max
π U ( ˆ P, π ) s.t.
g ( π ) + U ( ˆ P, π ) ≤ τ − ϵ 0 t .
Moreover, U ( ˆ P, π r ) ≤ T < ˜ κ.
Applying Lemma 2 with the baseline policy π 0 , we have
[Equation]
Let π min =
arg min π V
πP ∗ ,c .
By the definition of U and Assumption 1, we have
[Equation]
We again apply Lemma 2 with the feasible solution fixed as policy π min to conclude that
[Equation]
where ( i ) follows from Equation (13), and ( ii ) follows from the definition of ∆( c, τ ) .
Continuing this process, we get a sequence { x n } with recursive formula
[Equation]
and
[Equation]
Denote ∆( c, τ ) − ϵ 0 t by ˜∆ c .
Then, T ≤ ˜∆ c U / 2 < ˜ κ (˜∆ c − ˜ κ ), which implies that
[Equation]
Then, based on Lemma 20, { x n } converges to
[Equation]
Therefore, we conclude that
[Equation]
Let ˜ π = arg max π V
πP ∗ ,r ∗ .
By the definition of U ( π ) , we can compute the suboptimality gap of ¯ π as follows.
[Equation]
where ( i ) follows from the optimality of ¯ π ,
i.e. V ˜ π
ˆ P ϵ ,r ≤ V ¯ π ˆ P ϵ ,r .
Step 3: This step shows that SWEET can find an ϵ -optimal policy in the planning phase for any given reward r ∗ and under any constraint ( c ∗ , τ ∗ ) .
Let g
( π ) =
V π ˆ P,c ∗ .
Recall that
[Equation]
If g 0 ( π ∗ ) + U ( π ∗ ) ≤ τ ∗ , by the optimality of ¯ π , we immediately have
[Equation]
If g 0 ( π ∗ ) + U ( π ∗ ) > τ ∗ , then by the definition of U and Equation (14), we have
[Equation]
[Equation]
Let π = arg
min π V
πP ∗ ,c ∗ .
By Equation (14), we have
[Equation]
Let π γ be the Markov policy equivalent to the mixture policy γπ ∗ ⊕ (1 − γ )
π under the estimated model ˆ P .
Let ∆ c ∗ =
∆( c ∗ , τ ∗ ) =
τ ∗ − V πP ∗ ,c ∗ and γ = (∆ c ∗ − 3 U ) / ∆ c ∗ .
By the linearity of gand Equation (14), we have
[Equation]
where ( i ) follows from Equations (15) and (16).
This implies that π γ is a feasible solution of the optimization problem solved in the planning phase.
By the optimality of ¯ π and the linearity of V π ˆ P,r ∗ , we have
[Equation]
[Equation]
where ( i ) follows because γ = (∆ c ∗ − 3 U ) / ∆ c ∗ , and the last inequality follows from the normalization condition and Assumption 1.
Recall U ≤ ∆ min ϵ 5 .
Therefore, the suboptimality gap under ¯ π can be computed as follows.
[Equation]
where ( i ) follows from Equation (17).
A. P ROOF OF CONCAVITY OF THE TRUNCATED VALUE FUNCTION
In this subsection, we show that the truncated value function defined in Equation (4) is concave and continuous on the Markov policy space X .
These are crucial properties to be used for instantiating our theorem to Tabular-SWEET and Low-rank-SWEET.
Lemma 3 (Concavity of the truncated value function) .
Let π γ be the equivalent markov policy of γπ ⊕ (1 − γ )
π ′ under a transition model P .
Then,
[Equation]
In addition, ¯ V π γ P,u is continuous w.r.t.
γ ∈ [0 , 1] .
Moreover, if the utility function u satisfies the normalization condition, then the equality holds, i.e.,
[Equation]
Proof.
Recall that the truncated value function is defined recursively as follows:
[Equation]
with V α,πH +1 ,P,u ( s H +1 ) = 0 .
The continuity of the truncated value function is straightforward, since it is a composition of H continuous functions.
Therefore, we focus on the concavity part in the following analysis.
By Lemma 16, the following equality holds for any utility function u and time step h .
[Equation]
We then prove the claim by induction.
First, we note that when h = H + 1 ,
[Equation]
Assume it holds for step h
+ 1 ,
i
[Equation]
e.,
[Equation]
Then, for step h
[Equation]
by Jensen’s inequality
[Equation]
we have
[Equation]
where ( i ) follows from Equation (18) and the induction hypothesis.
Therefore, at step h = 1 , we have
[Equation]
If u satisfies the normalization condition, then Q πh,P,u
( s h , a h ) ≤ 1 holds for any π and h .
By the definition of the truncated value function, we have ¯ Q πh,P,u ( s h , a h ) =
Q πh,P,u ( s h , a h ) and ¯
V πh,P,u ( s h ) =
V πh,P,u
( s h ) , which implies that the truncated value function and the true value function are identical.
By Lemma 16, π γ introduces the same marginal probability over any state-action pair as the mixture policy γπ ⊕ (1 − γ ) π ′ .
Therefore, when u is normalized,
[Equation]
which completes the proof.
B A NALYSIS OF T ABULAR -SWEET
In this section, we first elaborate the Tabular-SWEET algorithm in Appendix B.1.
To provide the analysis for this algorithm, we first provide several supporting lemmas in Appendix B.2, and then prove Theorem 2 in Appendix B.3.
B. T HE T ABULAR -SWEET ALGORITHM
We first specify the parameters in Tabular-SWEET.
The detail of Tabular-SWEET is shown in
Algorithm 2.
Let U = min (cid:110) ϵ 2 , ∆ min 2 , ϵ ∆ min 5 , τ 4 , κ 16 (cid:111) , and T = ∆( c, τ ) U / 2 be the termination condition of TabularSWEET.
Let the maximum iteration number N be the solution of the following equation:
[Equation]
where β = log(2 SAH/δ ) + S log( e (1 + N )) .
Recall that the estimated model is computed by
[Equation]
where N ( n ) h ( s h , a h ) and N ( n ) h
( s h , a h , s h +1 ) denote the numbers of visits of ( s h , a h ) and ( s h , a h , s h +1 ) up to n -th episode, respectively.
Then, the exploration-driven virtual reward is defined as
[Equation]
where β 0 = 8 β .
The approximation error bound is a concave function of the truncated value function, defined as
[Equation]
Since ϵ 0 = t = 0 and ˜ κ = κ/ 2 , the safety set C ( n ) is given by
[Equation]
) ≥ τ − κ/ 2 , (cid:110) π :
V π ˆ P,c
+ U ( n ) ( π ) ≤ τ (cid:111) , otherwise.
Use π ( n − 1) to collect { s ( n )
, a ( n ) } ; D ← D ∪ { s ( n ) h , a ( n ) h , s ( n ) h +1 } Hh =1 ; 5:
Estimate ˆ P ( n ) ; update ˆ b ( n ) h , U ( n ) ( π and the empirical safe policy set C ( n ) (Equations (20) to
Solve π ( n ) = arg max π ∈C ( n ) U ( n ) ( π ) ; 7: if
C ( n ) (cid:12)(cid:12)(cid:12) > 1 and U ( n ) ( π ( n ) ) ≤ T then 8: (cid:16) n
, ˆ P
, ˆ b
(cid:17) ← n, ˆ P n
, ˆ b n
(cid:17) , break; 9: end if 10: end for 11: //
Planning: 12: Receive reward function r ∗ and safety constraint ( c ∗ , τ ∗ ) ; 13: Output: ¯ π = arg max π V π ˆ P ϵ ,r ∗ s.t.
V π ˆ P ϵ ,c ∗ + 4 (cid:113) ¯ V α H ,π ˆ P ϵ , ˆ b ϵ ≤ τ ∗ .
First, denote Var P h V πh +1 ,P ′ ,u ( s h , a h ) as the variance of value function V πh +1 ,P ′ ,u ( s h +1 ) , where s h + follows the distribution P h ( ·| s h , a h ) , i.e.,
[Equation]
Then, we have the following lemma.
Lemma 4 (Lemma 3 in M´enard et al.
Let ρ π ∗ ,h ( s h , a h ) be the marginal probability over state-action pair ( s h , a h ) induced by policy π under the true environment P ∗ .
Suppose the utility function u satisfies the normalization condition.
Denote
[Equation]
where β = log(3 SAH/δ ) + S log(8 e (1 + N )) , β 1 = log(3 SAH/δ ) , and β 2 = log(3 SAH/δ ) + log(8 e (1 + N )) .
Note that β ≥ β 2 ≥ β 1 .
Let E =
E
∩ E 1 ∩ E 2 .
Then, we have
P [ E ] ≥ 1 − δ.
The following lemma shows the relationship between the visitation counters N ( n ) h
( s h , a h ) and the pseudo-counter (cid:80) n − 1 m =
ρ π ( m ) ∗ ,h ( s h , a h ) , where ρ π ∗ ,h
( s h , a h ) is the marginal distribution on ( s h , a h ) induced by policy π under true model P ∗ .
Lemma 5 (Lemma 7 in Kaufmann et al.
(2021), Lemma 8 in M ´ enard et al.
On the event E , we have
[Equation]
In addition, we generalize Lemma 7 in M ´ enard et al.
(2021) from deterministic policies to randomized policies.
This is important for safe RL, as the optimal policy in constrained RL is possibly randomized.
Lemma 6 (Law of total variance with randomized policy) .
Given model P , policy π , and normalized utility function u , define another utility function σ h ( s h , a h ) as
[Equation]
Then, for any Markov policy π and h ∈ [ H ] , the following bound holds:
[Equation] [Equation]
In particular, when h = 1 , we have
[Equation]
where ρ πh ( s h , a h ) is the marginal distribution over state-action pair ( s h , a h ) induced by policy π under model P .
Proof.
First, we note that the statement is trivial for h = H + 1 since all Q-value functions are 0.
Then, we prove the result through induction.
Assume that at time step h + 1 ,
[Equation]
Then, at time step h
[Equation]
the LHS of Equation (24) can be computed as follows.
[Equation]
 
[Equation]
The term within the expectation in the third term equals 0 if we further condition it on s h
+1 , a h +1 , indicating that the third term is 0.
Therefore, from the assumption
[Equation]
we have
[Equation]
where ( i ) follows from Jensen’s inequality.
Thus, Equation (24) holds for all step h , and the proof is completed.
The following lemma is the key to ensure that Tabular-SWEET satisfies the termination condition.
Lemma 7.
On the event E , the summation of ¯ V α H ,π ( n ) ˆ P n , ˆ b ( n ) over any subset N ⊂ [ N ] scales in the order of log |N| , i.e.
(cid:88)
[Equation]
Proof.
First, similar to the truncated value function, we extend the definitions of value function to incorporate the additional factor α H .
Specifically, ∀ h ∈
[ H ] ,
[Equation]
and V α H ,π H
+1 ,P,u = 0 .
We then examine the difference between the truncated Q-value function defined with respect to model ˆ
P ( n ) and the Q-value function defined with respect to model P ∗ .
[Equation]
By Lemma 10 in M´enard et al.
[Equation]
we bound the first term as follows.
[Equation]
where ( i ) follows from √ 2 AB ≤ A/H + BH/
[Equation]
and ( ii ) is due to the truncated value function is at most 1 and Var( X ) ≤ E
[ X ] if X ∈ [0 , 1] .
Therefore
[Equation]
by combining the above two inequalities and taking expectation
[Equation]
we have,
[Equation]
Telescoping the above inequality from h = 1 to H and defining b ( n ) h ( s h , a h ) =
min (cid:110)
ˆ b ( n ) h
( s h , a h ) , 1 (cid:111) , we get
[Equation]
Therefore
[Equation]
if ρ π ( n ) ∗ ,h ( s h , a h ) is the marginal distribution over state-action pairs induced by exploration policy π ( n ) under the true model P ∗
[Equation]
we have
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
where ( i ) is due to Lemma 5, ( ii ) follows from Lemma 18, and ( iii ) follows the fact that ρ π ( m ) ∗ ,
h ( s h , a h ) ≤ 1 .
Therefore, (cid:88)
n ∈N ¯
[Equation]
B. P ROOF OF T HEOREM
Theorem 5 (Complete version of Theorem 2) .
Given ϵ, δ ∈ (0 , 1) , and safety constraint ( c, τ ) , let
U = min (cid:110) ϵ 2 , ∆ min 2 , ϵ ∆ min 5 , τ 4 , κ 16 (cid:111) , and T = ∆( c, τ ) U / 2 be the termination condition of TabularSWEET.
Then, with probability at least 1 − δ , Tabular-SWEET achieves the learning objective of safe reward-free exploration (Equations (1) and (2) ), and the number of trajectories collected in the exploration phase is at most O
[Equation]
where ι = log (cid:16) βHSA
∆( c,τ ) 2 U 2 + βHSAκ 2 (cid:17) , and β = log(2 SAH/δ ) + S log( e (1 + N )) .
Proof.
The proof of Theorem 2 mainly instantiates Theorem 1 by verifying that (a) U ( n ) ( π ) = 4 (cid:113)
¯ V α H ,π ˆ P ( n ) , ˆ b ( n ) is a valid approximation error bound for V π ˆ P ( n ) ,u , and (b) Tabular-SWEET satisfies the termination condition within N episodes.
The proof consists of three steps with the first two steps verifying the above two conditions and the last step characterizes the sample complexity.
Step 1: This step establishes the following lemma, which shows that U ( n ) ( π ) = 4 (cid:113) ¯ V α H ,π ˆ P ( n ) , ˆ b ( n ) is a valid approximation error bound.
Lemma 8.
With α H = 1 + 1 /H defined in Tabular-SWEET (Algorithm 2), on the event E , for any policy π and any utility normalized function u ,
[Equation]
[Table]
Proof.
Recall that
[Equation]
where β 0 = 8 β .
Define utility function u v as
[Equation]
Following Step 1 of Lemma 1 in M´enard et al.
(2021), we get
[Equation]
Next, we aim to show that
[Equation]
For that
[Equation]
let ˆ ρ π ( s h , a h ) be the marginal distribution over state-action pair ( s h , a h ) induced by model
ˆ P ( n ) and policy π .
Note that the truncated value function is a lower bound of the corresponding value function.
Thus
[Equation]
we can expand ¯ V α H [Equation] π ˆ P ( n ) ,u v as follows:
[Equation]
where ( i ) follows from the fact that (1 + 1 /H )
H ≤ e and ( ii ) follows from Cauchy-Schwarz inequality.
Note that in contrast to the optimistic policy, π could be a randomized policy in general.
By Lemma
[Equation]
we have
[Equation]
Meanwhile, if we define u bh ( s h , a h ) =
min
[Equation]
, which is obviously a normalized utility function, then, we have
[Equation]
where the last inequality follows from the facts that u bh ( s h , a h ) ≤ ˆ b ( n ) h ( s h , a h ) and α H > 1 .
Thus, we have Equation (26) established.
Combining Equations (25) and (26), we conclude that
[Equation]
where ( i ) is due to the fact that the truncated value function is at most 1.
Step 2: This step establishes the following lemma, which shows that Tabular-SWEET will terminate within N episodes.
Lemma 9.
On the event E , there exists n ϵ ∈
[ N ] such that (cid:12)(cid:12)(cid:12) C ( n ϵ ) (cid:12)(cid:12)(cid:12) > 1 and ¯ V α H ,π ( nϵ ) ˆ P ( nϵ ) , ˆ b ( nϵ ) ≤ T 2 / 16 , where N is defined in Equation (19) , and T is defined in Tabular-SWEET
(Algorithm 2).
Proof.
Denote N 0 = { n ∈
[ N ] : π ( n ) = π 0 } .
We first prove N 0 is finite.
Note that for all n ∈ N 0 ,
[Equation]
By Lemmas 7 and 8, we have
[Equation]
where the last inequality is due to Cauchy-Schwarz inequality.
Therefore, |N
| ≤
[Equation]
Then, we prove Lemma 9 by contradiction.
Assume ¯ V π ( n ) ˆ P ( n ) , ˆ b ( n ) > T 2 / 16 , ∀ n ∈ [ N ] \N 0 .
According to Lemma 7, we have
[Equation]
[-|N 
-]N 0 | + 1) , which implies that
[Equation]
This contradicts with the condition that N =[-
-]
[Equation]
Therefore, by noting that U ( n ) ( π ) = 4 (cid:113) ¯ V α H ,π ( n ) ˆ P ( n ) , ˆ b ( n ) , there exists n ϵ ∈
[ N ] such that the exploration phase under
Tabular-SWEET terminates.
Step 3:
This step analyzes the sample complexity as follows.
On the event E , since T = ∆( c, τ ) U / 2 , by Lemma 9, the sample complexity is at most
[Equation]
Note that n = c 0 log( c 1 n ) implies n ≤ 2 c 0 log( c 0 c 1 ) .
Thus,
[Equation]
where
[Equation]
Therefore, Tabular-SWEET terminates in finite episodes.
Besides, U ( π ) = 4 (cid:113) ¯ V α H ,π ˆ P ϵ , ˆ b ϵ .
On the event E , by Lemma 3, Lemma 8 and the concavity of √ x , U ( π ) is an approximation error function under ˆ P ϵ , and is concave and continuous on X .
We further note that κ/ 2(∆( c,τ ) − κ/ 2) 4∆( c,τ ) ≥ κ 16 due to the condition ∆( c, τ ) ≥ κ , which indicates that
T = ∆( c, τ ) U / 2 satisfies the requirement in Theorem 4.
Therefore, by Theorem 4, we conclude that with probability at least 1 − δ , the exploration phase of
Tabular-SWEET is safe and ¯ π is an ϵ -optimal policy subject to the safety constraint ( c ∗ , τ ∗ ) .
C A NALYSIS OF L OW - RANK -SWEET
In this section, we first elaborate the Low-rank-SWEET algorithm in Appendix C.1.
To provide the analysis for this algorithm, we first provide several supporting lemmas in Appendix C.2, and then prove Theorem 3 in Appendix C.3.
C. T HE L OW - RANK -SWEET ALGORITHM
We first specify the parameters adopted in Low-rank-SWEET, which is presented in Algorithm 3.
Let U = min (cid:110) ϵ 2 , ∆ min 2 , ϵ ∆ min 5 , τ 6 , κ 24 (cid:111) , and T = ∆( c, τ ) U / 3 be the termination condition of Lowrank-SWEET.
Recall that we set ϵ
=
κ/ 6 , t = 2 , and ˜ κ = κ/ 3 .
We define the maximum number of iterations N as
[Equation]
[Equation] where ζ = log (cid:0) 2 | Φ || Ψ | NH/δ (cid:1) , and β 3 is defined in Lemma 10.
Besides, we set ˜ A
= A/ϵ 0 and ˆ α = 5 (cid:113) β 3 ζ ( ˜ A
+ d 2 ) .
For ease of exposition, we introduce the following notation for an ( ϵ 0 , t ) -greedy version of policy π , denoted as G ϵ 0 H π , as follows:
[Equation]
where |H| = t .
Intuitively, G ϵ
H π follows π at time step h ∈ H with probability 1 − ϵ 0 and takes uniformly action selection with the probability ϵ 0 .
We also define
[Equation]
29) where Unif ( X 0 ) is a mixture policy that uniformly chooses one policy from the policy set X 0 ⊂ X .
We use G ϵ 0 H Π n to denote the ( ϵ 0 , |H| ) -greedy version of Π n .
C. S UPPORTING L EMMAS
We first characterize the following high probability event.
Lemma 10.
Denote
[Equation]
where λ = β 3 d log(2 NH | Φ | /δ )) and β 3 =
O (1) .
Define events E 0 and E 1 as
[Equation]
ˆ ϕ ( n ) h − 1 ( s, a ) (cid:13)(cid:13)(cid:13) ( U ( n ) h − 1 , ˆ ϕ ) −(cid:27) , where ζ = log (cid:0) 2 | Φ || Ψ |
NH/δ (cid:1) .
Denote E : =
E
∩ E 1 .
Then, P [ E ] ≥ 1 − δ .
Algorithm
1: Input: Constants ϵ 0 = κ/ 6 , ˜
A = A/ϵ 0 , and termination condition T .
2: // Exploration: 3: for n = 1 , . . .
, N do 4: for h = 1 , ..., H do 5:
Execute policy G ϵ 0 h − 1 ,h π ( n − 1) and collect data s ( n,h ), a ( n,h ), . .
, s ( n,h ) H , a ( n,h ) H 6:
D nh ← D
nh
∪ { s ( n,h ) h , a ( n,h ) h , s ( n,h ) h +1 } 7: end for 8:
Learn (ˆ ϕ ( n ) h , ˆ µ ( n ) h ) =
MLE ( D nh ) , and update ˆ P ( n ) according to Equation (5) 9:
Update empirical covariance matrix ˆ U ( n ) h according to Equation (6) 10:
Define exploration-driven reward function ˆ b ( n ) h ( · , · ) = min (cid:110)
ˆ α
∥ ˆ ϕ ( n ) h ( · , · ) ∥ (ˆ U ( n ) h ) − 1 , 1 (cid:111) .
Define U ( n ) ( π ) =
¯ V π ( n ) ˆ P ( n ) , ˆ b ( n ) + (cid:113) ˜
Aζ/n
C ( n ) L = 
 { π 0 } , if V π 0 ˆ P,c
+ U ( n ) ( π 0 ) ≥ τ − 2 κ/ 3 , (cid:110) π :
V π ˆ P,c
+ U ( n ) ( π ) ≤ τ − κ/ 3 (cid:111) , otherwise.
Solve π ( n ) = arg max π ∈C ( n ) L U ( n ) ( π ) , where C ( n ) L is defined in Equation (30).
13: if (cid:12)(cid:12)(cid:12) C ( n ) L (cid:12)(cid:12)(cid:12) > 1 and U ( n ) ( π ( n ) ) ≤ T then 14: (cid:16) n ϵ , ˆ P ϵ , ˆ b ϵ (cid:17) ← (cid:16) n, ˆ P ( n ) , ˆ b ( n ) (cid:17) , break 15: end if 16: end for 17: //
Planning: 18: Receive reward function r ∗ and constraint ( c ∗ , τ ∗ ) , 19: Output: ¯ π = arg max π V π ˆ P ϵ ,r ∗ s.t.
V π ˆ P ϵ ,c ∗ + ¯ V π ˆ P ϵ , ˆ b ϵ + (cid:113) ˜
Aζ n ϵ ≤ τ ∗ .
Proof.
By Corollary 2 in Appendix D, we have P [ E 0 ] ≥ 1 − δ/ 2 .
Further, by Lemma 39 in
Zanette et al.
(2020a) for the version of fixed ϕ
and Lemma 11 in Uehara et al.
(2021), we have P [ E 1 ] ≥ 1 − δ/ 2 .
Therefore, P [ E ] ≥ 1 − δ .
Based on Lemma 10, we can bound the exploration-driven virtual reward in Low-rank-SWEET as follows.
Corollary 1.
Given that the event E occurs, the following inequality holds for any n ∈ [ N ] , h ∈ [ H ] , [Equation]
[Equation]
where ˆ α = 5 (cid:113) β 3 ζ ( ˜ A + d 2 ) .
Proof.
Recall ˆ b ( n ) h
( s h , a h ) = min (cid:26) ˆ α (cid:13)(cid:13)(cid:13)
ˆ ϕ ( n ) h ( s, a ) (cid:13)(cid:13)(cid:13) (ˆ U ( n ) h ) − 1 ,(cid:27) .
Applying Lemma 10, we can immediately obtain the result.
The following lemma summarizes Lemmas 12 and 13 in Uehara et al.
and generalizes them to ϵ 0 -greedy policies.
We provide the proof for completeness.
Lemma 11.
Let P
h − 1 = ⟨ ϕ h − 1 , µ h − 1 ⟩ be a low-rank MDP model, and Π be an arbitrary and possibly a mixture policy.
Define an expected Gram matrix as follows:
[Equation]
Further, let f h − 1 ( s h − 1 , a h − 1 ) be the total variation distance between P ∗ h − 1 and P h − 1 at time step
h − 1 .
Suppose g ∈ S ×
A → R is bounded by B ∈ (0 , ∞ ) , i.e., ∥ g ∥ ∞ ≤ B .
Then, for h ≥ 2 , any policy π h ,
[Equation]
Proof.
We first derive the following bound:
[Equation]
where the inequality follows from Cauchy-Schwarz inequality.
We further expand the second term in the RHS of the above inequality as follows.
[Equation]
where ( i ) follows from the assumption that ∥ g ∥ ∞ ≤ B , ( ii ) follows from Jensen’s inequality, and that f h − 1 ( s h − 1 , a h − 1 ) is the total variation distance between P ∗ h − 1 and P h − 1 at time step
h − 1 .
For ( iii ) , note that G ϵ 0 h Π( ·| s h ) ≥ ϵ 0 /A
= 1 / ˜ A , which implies that π h ( ·| s
h ) ≤ 1 ≤ ˜ AG ϵ 0 h Π( ·| s h ) .
This finishes the proof.
Based on Lemma 11, we summarize three useful inequalities in the following lemma, which bridges the total variation f ( n ) h and the exploration-driven reward ˆ b ( n ) h .
Lemma 12.
Define
[Equation]
where λ = β 3 d log(2 NH | Φ | /δ )
Given that the event E occurs, the following inequalities hold for any iteration n : When h ≥ 2 ,
[Equation]
where
[Equation]
=
[Equation]
When h = 1 , E
[Equation]
Proof.
We start by showing Equation (34) as follows.
Given that the event E occurs, we have E
[Equation]
E
[Equation]
where ( i ) follows from Lemma 11 and the fact that f ( n ) h ( s h , a h ) ≤ 1 , ( ii ) follows from importance sampling at time step h − 2 , and ( iii ) follows from Lemma 10.
Equation (35) follows from the arguments similar to the above
[Equation] To obtain Equation (36), we first apply Lemma 11 and obtain
[Equation]
where we use the fact that ˆ b ( n ) h ( s h , a h ) ≤ 1 .
We further bound the term n E sh [Equation] [Equation] ˆ
[Equation] [Equation] s
[Equation]
, a h )) 2 ] as follows:
[Equation]
where ( i ) follows from Lemma
[Equation]
and we use tr( A ) to denote the trace of any matrix A .
Thus,
E
[Equation]
In addition
[Equation]
for h = [Equation] we have
E
[Equation]
and
E
[Equation]
E
[Equation]
n
[Equation]
where both ( i ) and ( ii ) follow from Jensen’s inequality and importance sampling.
The following lemma is key to ensure that Low-Rank-SWEET terminates in finite episodes.
Lemma 13.
Given that the event E occurs, the summation of the truncated value functions ¯ V π ( n ) ˆ P ( n ) , ˆ b ( n ) under exploration policies { π ( n ) } n ∈N is sublinear with respect to |N| for any N ⊂ [ N ] , i.e., the following bound holds:
[Equation]
Proof.
Note that ¯ V πh, ˆ P ( n ) , ˆ b ( n ) ≤ 1 holds for any policy π and h ∈
[ H ] .
We first have
[Equation]
which implies ¯ V
[Equation]
Applying the Equation (36) and Equation (37)
[Equation]
we obtain the following bound on the value function
V π n P ∗ , ˆ b ( n ) :
[Equation]
Similarly
[Equation]
we obtain
[Equation]
n
[Equation]
(cid:21) + (cid:115) ζ ˜ A n .
Then, taking the summation of V π n P ∗ , ˆ b ( n ) + f ( n ) over n ∈ N , we have (cid:88) n ∈N
[Equation]
+
[Equation]
n
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where ( i ) follows from Cauchy-Schwarz inequality and importance sampling
[Table]
and ( ii ) follows from
Lemma 19.
Hence, the statement of Lemma 13 is verified
[Table]
C. P ROOF OF T HEOREM
Theorem 6 (Restatement of Theorem 3) .
Given ϵ, δ ∈ (0 , 1) , and safety constraint ( c, τ ) , let
U = min (cid:110) ϵ 2 , ∆ min 2 , ϵ ∆ min 5 , τ 6 , κ 24 (cid:111) , and T = ∆( c, τ ) U / 3 be the termination condition of Low-rankSWEET.
Then, with probability at least 1 − δ , Low-rank-SWEET achieves the learning objective of safe reward-free exploration (Equations (1) and (2) ) and the number of trajectories collected in the exploration phase is at most
[Equation]
where
[Equation]
Proof.
The proof of Theorem 3 mainly instantiates Theorem 1 by verifying that (a) U ( n ) ( π ) =
¯ V π ˆ P ( n ) , ˆ b ( n ) + (cid:113) ˜
Aζ/n is a valid approximation error bound for V π ˆ P ( n ) ,u , and (b) Low-rank-SWEET satisfies the termination condition within N iterations.
The proof consists of three steps with the first two steps verifying the above two conditions and the last step characterizes the sample complexity.
Step 1: This step establishes that U ( n ) ( π ) =
¯ V π ˆ P ( n ) , ˆ b ( n ) + (cid:113) ˜
Aζ/n is a valid approximation error bound in Low-rank-SWEET.
Lemma 14.
For all n ∈ [ N ] , policy π and the normalized utility function u , given that the event E occurs, we have
[Equation]
Proof.
We first show that (cid:12)(cid:12)(cid:12)
[Equation]
Recall the definition of the truncated value functions ¯ V h, ˆ P ( n ) ,u ( s h ) and ¯ Q h, ˆ P ( n ) ,u ( s h , a h ) : ¯ Q π h, ˆ P ( n ) ,u
[Equation]
h , a h
[Equation]
u
[Equation]
a
[Equation]
We develop the proof by induction.
For the base case h =
H + 1 , we have
[Equation]
Assume
[Equation]
) holds for any s h +1 .
Then, from Bellman equation [Equation] we have,
[Equation]
where ( i ) follows from ∥ ˆ P ( n ) h ( ·| s h , a h ) − P ∗ h ( ·| s h , a h ) ∥
1 = f ( n ) h
( s h , a h ) and the assumption that u is normalized
[Equation]
and ( ii ) follows from the induction hypothesis
[Equation]
Then, by the definition of ¯ V πh
[Equation]
ˆ P ( n )
[Equation]
u ( s h )
[Equation]
we have
[Equation]
where ( i ) follows because ˆ Q πh, ˆ P ( n ) ,u ( s h , a h ) − Q πh,P ∗ ,u ( s h , a h ) > − 1 , and ( ii ) follows from Equation (38).
Therefore, by induction, we have
[Equation]
Then, we show that ¯ V π ˆ P ( n
[Equation]
By Equation (34) and the fact that the total variation distance is upper bounded by 1, with probability at least 1 − δ/ 2 , we have
[Equation]
Similarly, when h = 1 ,
[Equation]
Based on Corollary 1, Equation (39) and α = 5ˆ α , we have
[Equation]
[Equation]
Assume that E ˆ P ( n ) ,π
[Equation]
) holds for step h
+ 1 .
Then, by
Jensen’s inequality
[Equation]
we obtain
[Equation]
where ( i ) follows from Equation (41)
[Equation]
and ( ii ) is due to the induction hypothesis
[Equation]
By induction, we conclude that
[Equation]
Combining Step 1 and Step 2, we conclude that
[Equation]
Step 2: This step shows that Low-rank-SWEET terminates in finite episodes.
Lemma 15.
On the event E , there exists n ϵ ∈
[ N ] such that ¯
V π nϵ ˆ P ( nϵ ) , ˆ b ( nϵ ) + (cid:113) ˜
Aζ/n ϵ ≤ T , where N is defined in Equation (27) .
Let N 0 = (cid:26) n :
[Equation]
We first show that N 0 is a finite set.
Note that, n ∈ N 0 implies that ¯ V π 0 ˆ P ( n ) ,c
+ ¯ V π 0 ˆ P ( n ) , ˆ b ( n [Equation] + (cid:113) ˜
Aζ/n > τ − 2 κ/ 3 , and π ( n ) = π 0 .
Then, we have,
[Equation]
where ( i ) is due to Lemma 14 and the ( ii ) follows from Lemma 13.
Therefore, we have
[Equation]
Next, we prove the existence of n ϵ via contradiction.
Assume ¯ V π ( n ) ˆ P ( n ) , ˆ b ( n
[Equation]
+ (cid:113) ˜
Aζ n > T , ∀ n ∈ [ N ] / N 0 .
By Lemma 13, we have
[Equation]
which implies
[Equation]
This contradicts with the fact that N = 2 10 β 3 H 2 d 4 ˜ A 2 ζ
[Equation]
[Equation]
Step 3:
This step analyzes the sample complexity of Low-rank-SWEET as follows.
Given that the event E occurs, since T = ∆( c, τ ) U / 3 and ˜ A = A/ϵ 0 = 6 A/κ , by Lemma 15, the number of iterations is at most
[Equation]
Note that n = c 0 log 2 ( c 1 n ) implies n ≤ 4 c 0 log 2 ( c 0 c 1 ) .
Thus,
[Equation]
where ι = log
[Equation]
Since there are H episodes in each iteration, the sample complexity is at most
[Equation]
Thus, Low-rank-SWEET terminates in finite episodes.
Note that U ( π ) =
¯ V π ˆ P ϵ , ˆ b ϵ + (cid:113) ˜
Aζ/n ϵ .
Given that the event E occurs, by Lemma 3 and Lemma 14,
U ( π ) is an approximation error function under ˆ P ϵ , and is concave and continuous on X .
We further note that κ/ 3(∆( c,τ ) − 2 κ/ 3) 4(∆( c,τ ) − κ/ 3) ≥ κ 24 , and ∆( c, τ ) − κ/ 3 ≥ 2∆( c, τ ) / 3 due to the condition ∆( c, τ ) ≥ κ , which implies that T = ∆( c, τ ) U / 3 satisfies the requirement in Theorem 4.
Therefore, using Theorem 4, we conclude that with probability at least 1 − δ , the exploration phase of Low-rank-SWEET is safe and ¯ π is an ϵ -optimal policy subject to the constraint ( c ∗ , τ ∗ ) .
D A UXILIARY LEMMAS
We first provide the following property of a mixture policy and its equivalent Markov policy for completeness.
Lemma 16 (Theorem 6.1 in Altman (1999)) .
Given a model P , any Markov policies π, π ′ ∈ X , and γ ∈ [0 , 1] , there exists π γ ∈ X that is Markov and equivalent to the mixture policy γπ ⊕ (1 − γ ) π ′ .
Let ρ πh ( s h ) and ρ πh ( s h , a h ) be the marginal distributions over the state and the state-action pairs induced by π under P , respectively.
Then, the following statements hold:
• ρ π γ h ( s h )
=
γρ πh ( s h ) + (1 − γ ) ρ π ′ h
( s h ) , • ρ π γ h
( s h , a h ) =
γρ πh ( s h , a h ) + (1 − γ ) ρ π ′ h
( s h , a h ) , • π γ ( a h | s h )
=
ρ π γ h ( s h , a h ) /ρ
π γ h ( s h ) , •
V π γ P,u = γV πP,u + (1 − γ )
V π ′ P,u holds for any utility function u .
Next, we present the estimation error of MLE in the n -th iteration at step h , given state s and action a , in terms of the total variation distance, i.e. f ( n ) h ( s, a ) =
(cid:13)(cid:13)(cid:13) ˆ P ( n ) h ( ·| s, a ) − P ∗ h ( ·| s, a ) (cid:13)(cid:13)(cid:13) 1 .
By Theorem 21 in Agarwal et al.
(2020), we are able to guarantee that under all exploration policies, the estimation error can be bounded with high probability.
Lemma 17.
(MLE guarantee).
Given δ ∈ (0 , 1) , we have the following inequality holds for any n ∈ [ N ] , h ∈
[ H ] with probability at least 1 − δ/ 2 :
[Equation]
where ζ := log (cid:0) 2 | Φ || Ψ | NH [Equation] δ (cid:1) .
Dividing both sides of the inequality in Lemma 17 by n , we have the following corollary hold, which is intensively used in the analysis.
Corollary 2.
Given δ ∈ (0 , 1) , the following inequality holds for any n, h ≥ 1 with probability at least 1 − δ/ 2 :
[Equation]
where Π n and G ϵ
h Π n are defined in Equation (29) and Equation (28) , respectively.
Then, we present two critical lemmas which ensure the summation of the approximation errors grows sublinearly in Tabular-SWEET and Low-rank-SWEET.
Lemma 18 (Lemma 9 in M´enard et al.
Suppose { a n } ∞ n =0 is a sequence with a n ∈ [0 , 1] , ∀ n .
Let S n = max { 1 , (cid:80) nm =0 a m } .
Then, the following inequality holds:
[Equation]
Lemma 19 (Elliptical potential lemma: Lemma B.3 in He et al.
Consider a sequence of d × d positive semidefinite matrices X 1 , . .
, X N with tr( X n ) ≤ 1 for all n ∈ [ N ] .
Define M 0 = λ
I and M n = M n − 1 + X n .
Then, ∀N ⊂ [ N ] ,
[Equation]
Finally, the following lemma is used in Theorem 4.
Lemma 20.
Given a, b > 0 , define a positive sequence { x n } n ≥ 1 recursively by
[Equation]
If a 2 > 4 b and x 1 ∈ [ a −√ a 2 − 4 b, a + √ a 2 − 4 b) , then, { x n } converges to a −√ a 2 − 4 b.
Proof.
Step 1.
We first show that x n ∈ [ a −√ a 2 − 4 b, a [Equation] √ a 2 − 4 b) .
This is true for n = 1 , as x 1 ∈ [ a −√ a 2 − 4 b,
[Equation]
− [Equation] b) .
Assume that x n − 1 ∈ [ a −√ a 2 − 4 b, a + √ a 2 − 4 b) .
Then, with simple algebra, we can show that
[Equation]
Step 2.
We show that { x n } is a non-increasing sequence.
Indeed, from Step 1, we have
[Equation]
Therefore, x n +1 ≤ x n holds for all n ≥ 1 .
Combining Steps 1 and 2, we conclude that there exists a limit of the sequence { x n } , denoted by x ∗ .
By the recursive formula, x ∗ must be a solution to the following equation
[Equation]
Since x ∗ ≤ x 1 < a + √ a 2 − 4 b, by solving the above equation, we conclude that
[Equation]