Provably Efficient Model-Free Constrained RL with Linear Function Approximation
Abstract
We study the constrained reinforcement learning problem, in which an agent aims to maximize the expected cumulative reward subject to a constraint on the expected total value of a utility function.
In contrast to existing model-based approaches or model-free methods accompanied with a ‘simulator’, we aim to develop the first model-free , simulator-free algorithm that achieves a sublinear regret and a sublinear constraint violation even in large-scale systems.
To this end, we consider the episodic constrained Markov decision processes with linear function approximation, where the transition dynamics and the reward function can be represented as a linear function of some known feature mapping.
We show that ˜
O ( √ d 3 H 3 T ) regret and ˜ O ( √ d 3 H 3 T ) constraint violation bounds can be achieved, where d is the dimension of the feature mapping, H is the length of the episode, and
T is the total number of steps.
Our bounds are attained without explicitly estimating the unknown transition model or requiring a simulator, and they depend on the state space only through the dimension of the feature mapping.
Hence our bounds hold even when the number of states goes to infinity.
Our main results are achieved via novel adaptations of the standard LSVI-UCB algorithms.
In particular, we first introduce primal-dual optimization into the LSVI-UCB algorithm to balance between regret and constraint violation.
More importantly, we replace the standard greedy selection with respect to the state-action function in LSVI-UCB with a soft-max policy.
This turns out to be key in establishing uniform concentration for the constrained case via its approximation-smoothness trade-off.
Finally, we also show that one can achieve an even zero constraint violation for large enough T by trading the regret a little bit but still maintaining the same order with respect to T .
Introduction
In many practical applications of online reinforcement learning (RL) (e.g., financial regulations, safety), there exist additional constraints on the learned policy in the sense that it also needs to ensure that the expected total utility (cost, resp.) exceeds a given threshold (is below a threshold, resp.).
Constrained RL problem is formulated as a constrained Markov Decision Process (CMDP), in which the celebrated exploration-exploitation trade-off in online RL becomes more challenging due to the additional need to find a balance between reward regret and constraint violations.
To develop online sample-efficient algorithms for CMDPs, prior works have largely resorted to model-based approaches, where the control policy is constructed based on a learned model [1–7].
However, due to the explicit estimation and storage of the unknown transition model, model-based approaches often lead to large time and space complexities.
These issues are exacerbated in the large state space.
Hence, there are several recent works starting to investigate model-free algorithms for
CMDPs, which directly update the value function or the policy without first estimating the model [8– 10].
However, all of these works consider an easier setting compared to standard RL in that they assume access to a simulator [11] (a.k.a.
a generative model [12]), which is a strong oracle that allows the agent to query arbitrary state-action pairs and return the reward and the next state, hence greatly alleviating the intrinsic difficulty of exploration in RL.
To the best of our knowledge, [13] were the first to study model-free and simulator-free algorithms for CMDPs.
The above mentioned work on model-free algorithm considers the finite-state tabular setting and the regret scales polynomially with the number of states.
Thus, the result would not be useful for large-scale RL applications where the number of states could even be infinite.
To address this curse of dimensionality, modern RL has adopted function approximation techniques to approximate the (action-)value function of a policy, which greatly expands the potential reach of RL, especially via deep neural networks.
However, little is known for the performance guarantee of model-free algorithms in CMDPs beyond tabular settings, even in the case of linear function approximation.
Motivated by this, we are interested in the following question:
Can we achieve provably sample-efficient and model-free exploration for CMDPs beyond tabular settings (without a simulator)?
Contribution.
To answer the
above question we consider
the episodic CMDPs with linear function approximation, where the transition dynamics and the reward function can be
represented
as a linear function of some
known feature mapping
Our main
contributions
are as follows
• We show that with a proper parameter choice, our proposed algorithm achieves ˜ O ( √ d 3 H 3 T ) regret and ˜ O ( √ d 3 H 3 T ) constraint violation bounds with a high probability, where d is the dimension of the feature mapping, H is the length of the episode, and T is the total number of steps.
We also show that it is in fact possible to achieve zero constraint violation by trading the regret a little bit while maintaining the same order with respect to T .
• Our bounds are attained without explicitly estimating the unknown transition model or requiring a simulator , and they depend on the state space only through the dimension of the feature mapping.
To
the best of knowledge, these sub-linear bounds are the first results for model-free, simulator-free online RL
algorithms
for CMDPs with function approximations.
We even improve the bound ( ˜ O ( T 0 . 8 ) ) proposed in the model-free finite state tabular setting by [13] (Table 1).
• We combine the primal-dual algorithm with the classic model-free, simulator-free LSVI-UCB algorithm [14] to balance between regret and constraint violations.
This naturally leads to the construction of a new composite state-action function (i
e., Q -function), which is the sum of the Q -function for the reward and the Q -function for the utility weighted by the dual variable.
Due to this new type of Q -function in CMDPs, a key challenge arises when establishing the value-aware uniform concentration, which lies at the heart of the performance analysis of model-free exploration.
More specifically, the standard greedy selection with respect to this composite Q -function fails in finding non-trivial covering number for the function class of individual value functions (i
e., V -function) for the reward and the utility respectively.
To address this fundamental issue, we instead adopt a soft-max policy by utilizing its nice property of approximation-smoothness trade-off via its parameter, i
e., temperature coefficient.
1.1 Related Work
Model-based RL algorithms have been proposed for the CMDP [1–3, 5–7].
Apart from [7], the rest considered tabular set-up.
In the tabular model-based set-up, the best known regret and constraint violations achieved are ˜ O
( (cid:112) |S| 2 |A| T ) where |S| and |A| are the dimensions of the state and action spaces respectively.
Hence, such results can not cope up with the large state space.
[7] considered linear kernel MDP whereas we consider linear MDP.
These two are not the same in general.
We describe the differences with the algorithm proposed in [7] in Section 4.1.
Model-free RL algorithms have also been proposed [8–10] to solve CMDP.
However, all of the above require a generator model, which simulates from any state and action.
[13] proposed a ‘triple-Q’
[Table]
algorithm which does not require a ’simulator’.
However
[Table]
it only considered tabular setting.
The regret bound shown in [13] is ˜ O
( T 0 . 8 ) which is far from the optimal for the model-based case ˜ O ( √ T ) .
Please see Table 1 to see our contribution compared to the state-of-the-art approaches.
[15] proposed a RL algorithm for the scenario where a constraint needs to be satisfied at each step of an episode.
We consider a constraint where the cumulative utility over the length of the episodes must exceed a threshold.
Hence, the set of constraints is fundamentally different.
The authors in [15] also assumed that a safe-action is known for each state which we do not assume in our setting.
Problem Formulation
We consider an episodic constrained MDP, denoted by ( S , A , P , H, r, g ) where S is the state space,
A is the action space, H is the fixed length of each episode, P = { P h } Hh =1 is a collection of transition probability measures,
r = { r h } Hh =1 is a collection of reward functions, and g =
{ g h } Hh = is a collection of utility functions.
We assume that S is a measurable space with possibly infinite number of elements, A is a finite action set.
P h ( ·| x, a ) is the transition probability kernel which denotes the probability to reach a state when action a is taken at state x .
r h : S ×
A → [0 , 1] , and g h : S ×
A → [0 , 1] and are assumed to be deterministic.
However, we can readily extend to settings when r h and g h are random.
Each episode k ∈ [ K ] starts with the fixed state x 1 .
It can be readily generalized to the setting where x 1 is drawn from a distribution.
Then at each step h ∈ [ H ] in episode k , the agent observes state x kh ∈ S , picks an action a kh ∈ A , receives a reward r h ( x kh , a kh ) , and a utility g h ( x kh , a kh ) .
The
MDP evolves to x kh +1 that is drawn from P h ( ·| x kh , a kh ) .
The episode terminates at step H + 1 .
Without loss of generality, we assume that r H +1 =
g H
+
= 0 .
In this paper, we consider the challenging scenario where the agent only observes the bandit information r h ( x kh , a kh ) and g h ( x kh , a kh ) at the visited state-action pair ( x kh , a kh ) .
The policy-space of an agent is ∆( A|S , H ) ; {{ π h ( ·|· ) } Hh =1 : π h ( ·| x ) ∈ ∆( A ) , ∀ x ∈ S , and h ∈
[ H ] }.
Here ∆( A ) is the probability simplex over the action space.
For any x kh ∈ S , k ∈ [ K ] , and h ∈
[ H ] , π h,k ( a kh | x kh ) denotes the probability that the action a kh ∈ A is taken at episode k when the state is x kh .
Let V πr,h ( x ) denote the expected value of the total reward function starting from step h and state x when the agent selects action using the policy π = { π h } Hh =
[Equation]
where E is taken with respect to the policy π and the transition probability kernel P .
Let Q πr,h ( x, a ) denote the expected value of the total reward starting from step h and the state-action pair ( x, a ) and follows the policy π as
[Equation]
Similarly, we define the value function for the utility V πg,h ( x ) , and the action-value function for the utility Q πg,h ( x, a ) .
We denote V πj,h ( x ) , and Q πj,h ( x, a ) for j = r, g .
Definition 1.
For brevity, we denote P h V πj,
h +1 ( x, a ) =
E x ′ ∼ P h ( ·| x,a ) V πj,
h +1 ( x ′ ) for j = r, g .
Using this notation, the Bellman’s equation associated with the policy π becomes
[Equation]
Note that
[Equation]
where
[Equation]
The objective of the learning agent is to find an optimal solution of the following problem
[Equation]
Note that even though we have only once constraint, it can be readily generalized to the scenario with multiple constraints.
In order to avoid trivial solutions, we consider b ∈ (0 , H ] .
We denote the optimal policy as π ∗ which solves the above optimization problem.
Since π ∗ is obtained by having complete information, it is also denoted as the best policy in the hindsight .
Without any constraint information a priori, an agent can not know the policies that satisfy the constraint.
Instead, we allow the policy to violate the constraint and minimize the regret while minimizing the total constraint violations over the K episodes.
Such an approach is also considered in the existing literature [1, 7, 9].
We now define the performance metric which we seek to minimize.
Performance Metric.
Let the policy employed by the agent at episode k be π k
= [ π 1 ,k , . .
, π h,k , . . .
, π H,k ] T .
The performance metric we are considering is the following
[Equation]
where [ z ] + = max { z, 0 } .
The regret is defined as the difference between the total reward value by following the optimal policy π ∗ , and the total reward value obtained by following agent’s policy π k at episode k over K episodes.
The constraint violation is defined as the difference between the threshold value Kb and the total utility function attained by following the policies over all the episodes K .
Linear Function Approximation.
To handle a possible large number of states, we consider the following linear MDPs.
Assumption 1.
The CMDP is a linear MDP with feature map ϕ : S ×
A → R d , if for any h, there exists d unknown signed measures µ h = { µ 1 h , . .
. , µ dh } over S such that for any ( x, a, x ′ ) ∈
S ×
A × S ,
[Equation]
(6) and there exists vectors θ r,h , θ g,h ∈ R d such that for any ( x, a ) ∈ S × A ,
[Equation]
Assumption 1 adapts the definition of linear MDP [14, 16] to the constrained case.
By the above definition, the transition model, the reward, and the utility functions are linear in terms of feature map ϕ .
We remark that despite being linear, P h ( ·| x, a ) can still have infinite degrees of freedom since µ h ( · ) is unknown.
Note that tabular MDP is part of linear MDP [14].
Note
that [7, 17] studied another related concept known as linear kernel MDP.
In the linear kernel
MDP, the transition probability is given by P h ( x ′ | x, a ) = ⟨ ψ ( x ′ , x, a ) , θ h ⟩ .
In general, linear MDP and linear kernel MDPs are two different classes of MDP [17].
Similar to Proposition 1 in [14], we can show that for a linear MDP and for any policy π there exists { w πj,h } Hh =1 such that Q πj,h ( x, a ) = ⟨ w πj,h , ϕ
( x, a ) ⟩ for any ( x, a, h ) ∈ S
×
A ×
[ H ] .
We, thus, focus on linear action-value function.
Dual problem and Slater’s Condition.
We first introduce few notations which we will use throughout this paper.
Definition 2.
V π,Yh ( · ) =
V πh,r ( · ) + Y V πh,g ( · ) , and Q π,Yh ( x, a ) =
Q πr,h ( x, a ) + Y Q πg,h ( x, a ) , where
Y is the dual variable.
Thus, V π,Yh ( · ) and Q π,Yh ( · , · ) are respectively the composite value function and Q -functions respectively.
We can cast the problem (4) as a saddle point problem max π min Y L ( π, Y ) where
L ( π, Y ) =
V πr, 1 ( x 1 ) + Y ( V πg, 1 ( x 1 ) − b ) =
V π,Y− Y b , where π is the primal policy and Y is the dual variable.
However, the lagrangian is non-concave in π [18] even though it is convex in Y .
Nevertheless, the strong duality holds [19].
Hence, there exists optimal dual variable Y ∗ , such that max π L ( π, Y ∗ ) will correspond to the optimal reward value function.
We assume the following slater’s condition in this paper.
Assumption 2 (Slater’s Condition) .
There exists γ > 0 , and ¯ π ∈ ∆( A|D , H ) , such that V ¯ πg, 1 ( x 1 ) ≥ b + γ ,
Lemma 1 (Boundedness of Y ∗ ) .
The optimal dual-variable Y ∗ ≤ V π
[Equation]
The slater’s condition is mild in practice and commonly adopted in previous works [7, 1, 20].
We use the properties of the slater’s condition to bound the performance of our proposed algorithm.
Definition 3.
We set ξ = 2 H/γ .
Our Approach
1: Initialization:
Y 1 = 0 , w j,h = 0 , ξ = 2 H/γ , α = log( |A| )
K 2(1 + ξ + H ) , η = ξ/ √ KH 2 , β =
C 1 dH (cid:112) log(4 log |A| dT/p ) 2: for episodes
k = 1 , . .
, K do 3:
Receive the initial state x k 1 .
4: for step
h
= H, H − 1 , . .
, 1 do 5: Λ
kh ← (cid:80) k − 1 τ =1 ϕ
( x τh , a τh ) ϕ
( x τh , a τh ) T
+ λ I 6: w kr,h ← (Λ kh ) − 1 [ (cid:80)
k − 1 τ =1 ϕ
( x τh , a τh )
[ r h ( x τh , a τh ) + V kr,h +
( x τh +1 )]] 7: w kg,h ← (Λ kh ) − 1 [ (cid:80)
k − 1 τ =1 ϕ
( x τh , a τh )
[ g h ( x τh , a τh ) + V kg,h +
( x τh +1 )]] 8:
Q kr,h ( · , · ) ← min {⟨ w kr,h , ϕ ( · , · ) ⟩ + β ( ϕ ( · , · ) T (Λ kh ) − 1 ϕ ( · , · )) 1 / 2 , H } 9:
Q kg,h ( · , · ) ← min {⟨ w kg,h , ϕ ( · , · ) ⟩ + β ( ϕ ( · , · ) T (Λ kh ) − 1 ϕ ( · , · )) 1 / 2 , H } 10: π h,k ( a |· ) = exp( α ( Q kr,h ( · , a ) + Y k Q kg,h ( · , a )))
(cid:80) a exp( α ( Q kr,h ( · , a ) + Y k Q kg,h ( · , a ))) 11:
V kr,h ( · ) =
(cid:80) a π h,k ( a |· ) Q k r,h ( · , a ) 12:
V kg,h ( · ) =
(cid:80) a π h,k ( a |· ) Q kg,h ( · , a ) 13: for step h = 1 , . . .
, H do 14:
Compute Q kr,h ( x kh , a ) , Q kg,h ( x kh , a ) , π ( a | x kh ) for all a .
Take action a kh ∼ π h,k ( ·| x kh ) and observe x kh +1 .
Y k +1 =
max { min { Y k + η ( b − V kg, 1 ( x 1 )) , ξ } , 0 }
We now describe our proposed algorithm in Algorithm 1.
This algorithm is based on the primal-dual adaptation of the LSVI-UCB [14].
For a given dual variable, the primal policy is updated, and then the dual value is updated based on the estimated utility value function.
At each episode, the algorithm consists of three parts.
The first part (Steps 4-12) consists of updating the parameters w k r,h , w k g,h and Λ kh which are used to update the Q kj,h and V kj,h at episode k .
Λ kh is the Gram-matrix for the regularized least square problem (see Eqn.
(8), later).
Note that the Steps 8-12 are not evaluated for each state, rather, they are evaluated only for the encountered states till episode k − 1 .
Hence, we do not need to iterate over potentially infinite number of states.
For the first episode, since k − 1 =and τ = 1 , we have w kj,h = 0 , ∀ j and Λ kh
= λ
I .
We note that Q kj,H +1 ( · , · ) = 0 for j = r, g .
The value functions are updated (Steps 11-12) based on Q function and the policy.
The policy is based (Step 10) on a soft-max policy unlike the greedy one in the unconstrained case [14].
Soft-max policy S OFT -M
AX α
( X ) =
{ S OFT -M
AX iα ( X ) } |A| i =1 for any vector X ∈ R |A| is a |A| -dimensional vector with parameter α where the i -th component
[Equation]
At step h , π h,k ( a | x ) is computed based on the soft-max policy on the composite Q -function vector { Q kr,h ( x, a ) + Y k Q kg,h ( x, a ) } a ∈A where Y k is the lagrangian multiplier.
When α = ∞ , this becomes equal to the greedy policy.
The second part (Steps 13-15) is the execution of the soft-max policy based on the composite Q -value for the encountered state x kh .
Q function and Value function Estimation.
We need to estimate the value-function and Q -function with respect to the policy π k .
However, there are challenges.
We do not know P h in Bellman’s equation (3), rather P h
V
π k j,h +1 should be replaced by the empirical samples.
Further, in the large state space, we can not iterate over all ( x, a ) .
Rather, we parameterize Q π ∗ j,h ( · , · ) by a linear form ⟨ w kj,h ( · , · ) , ϕ ( · , · ) ⟩ .
The intuition is to obtain w kj,h from the Bellman’s equation using the regularized least-square regression.
We obtain w kj,h for j = r, g according to the following equation
[Equation]
[Equation]
Then, an additional bonus term β ( ϕ ( · , · ) T (Λ kh ) − 1 ϕ ( · , · )) 1 / 2 is added as in [14].
β is constant which we will characterize in the next section.
Such an additional term is used for upper confidence bound in LSVI-UCB [14].
The same additional term is used for both Q kr,h and Q kg,h .
Note the difference with the LSVI-UCB, here, we need to estimate the value function corresponding to the soft-max policy π k where in LSVI-UCB, a greedy policy corresponding to the Q -function is used.
Policy.
We update a soft-max policy which selects actions according to the estimated ‘composite’
Q -function at the k -th episode.
The reason behind using the soft-max policy instead of a greedy policy will be apparent in the next section when we state the main results and the proof ideas.
Note from the strong duality, for optimal dual variable Y ∗ , optimal primal policy π ∗ maximizes the composite value function V π ∗ ,Y ∗ (Definition 2).
Thus, the optimal policy should be a greedy one based on this optimal dual value Y ∗ .
However, the greedy policy is not Lipschitz, hence, it does not provide uniform concentration bound for each individual value function, an essential step in the regret bound (Section 4.2).
Hence, compared to the unconstrained scenario, we need more exploration in the policy space where the apparent reason is that we do not know the optimal dual variable beforehand.
Since we use the soft-max policy, there is a gap compared to the optimal value even when the lagrangian multiplier Y k becomes equal to Y ∗ .
However, if α in the soft-max policy also scales with K , then we can bound the gap from the optimal value function (Section 4.2).
Dual Update.
To infer the constraint violation, we estimate V k g, 1 for V π k g, 1 .
We update the lagrangian multiplier Y k by moving towards minimizing the lagrangian L ( π, Y ) over Y ≥ 0 in line 16, where η > 0 is a step-size and ξ is the upper bound on the dual variable such that optimal dual variable Y ∗ is contained within [0 , ξ ] .
The dual update is similar to the step described in [7, 1].
The dual update works as a trade-off between the reward maximization and the constraint violation reduction.
If b − V kg, 1 ≥ 0 , that means with a high probability, the constraint will be violated for the policy π k .
Hence, the dual value is increased in order to focus on minimizing the constraint violation.
Otherwise, the agent tries to maximize the reward value function.
Space and Time Complexities.
We remark that Algorithm 1 only needs to store Y k , w kr,h , w kg,h , r h ( x kh , a kh ) , g h ( x kh , a kh ) , Λ kh , and { ϕ ( x kh , a ) } a ∈A for
all ( h, k ) ∈
[ H ] ×
[ K ] , hence, it takes O ( d 2 H + d A T ) space.
When we compute (Λ kh ) − 1 using Sherman-Morrison formula, the computation of V kj,h +1 is dominated by computing Q kj,h +1 and the policy π k .
Hence, it takes O ( d 2 A T ) time.
Note that since our approach is model-free and we do not need to evaluate integrals as in [7] in order to estimate P
h V
kh +1 .
Note that both η and α use the knowledge of K .
In case, K is unknown, one can use the “doubling trick" [21] which will only scale the regret and constraint violation by a constant factor.
Analysis
We now state the main result.
We prove that Algorithm 1 achieves regret and constraint violation which are sublinear in T =
KH where T is the total number of steps.
4.1 Main Results
Theorem 1.
Fix p > 0 .
If we set λ = 1 , β = C 1 dH √ ι in Algorithm 1 where ι = log(log( |A| )4 dT/p ) for some absolute constant C 1 .
With probability (1 − p ) ,
[Equation]
for some absolute constants C , and C ′ .
We remark the difference with the existing results.
Since ξ = 2 H/γ (by Definition 3), our result indicates that our approach obtains ˜ O ( √ d 3 H 3 T ) regret and the same order of constraint violation where ˜ O absorbs logarithmic factor on T .
The regret and constraint violation are sub-linear in T , and similar dependence is observed in [1, 7].
Also note that compared to the unconstrained case [14], there is an additional log( |A| )) factor in the value of ι which arises because we use soft-max policy instead of the greedy policy which adds to the covering number.
The regret and constraint violation do not depend on the dimension of the state space, rather, it depends on the dimension of the feature space.
To the best of our knowledge this is the first result which shows both ˜ O ( √ T ) regret and constraint violation in the model-free set up (tabular or linear) without requiring a simulator.
Comparison with [7] : Compared to [7] which also considers linear function approximation (however, it considers linear kernel MDP rather linear MDP)
we improve the result in [7] by a factor of H .
Second, compared to [7], which is a model-based policy-based algorithm, ours is a model-free value-based algorithm.
Due to this, the above uniform concentration challenge does not exist in [7].
Moreover, our model-free algorithm also enjoys an easy implementation and improved computation efficiency since it does not estimate the next step expected value function as in [7] which requires an integration oracle to compute a d -dimensional integration at every step.
[7] also needs to store the previous policies and estimated value functions, hence, it needs O ( T ) additional space complexity.
We have an additional √ d factor in front of the regret and constraint violation.
Similar difference in regret is also observed between the model-based linear kernel unconstrained MDP [22] and model-free linear unconstrained MDP [14] even in the unconstrained case .
Similar to the discussion in Section 3.1 on [23], our result directly translates to a sample complexity guarantee (or, PAC guarantee).
For example, we can learn a policy π such that V π ∗ r, 1 ( x 1 ) − V πr, 1 ( x 1 ) ≤ ϵ , and b − V πg, 1 ( x 1 ) ≤ ϵ after ˜ O ( d 3 H 4 /ϵ 2 ) number of samples.
Here, the policy π is obtained after running Algorithm 1 for ˜ O
( d 3 H 3 /ϵ 2 ) number of episodes, and then selecting policy π k with probability 1 /K for any k ∈
[ K ] .
Recently, [24] proposed an algorithm with provable sample complexity guarantee for linear CMDP.
However, the regret and violation guarantees are different from the sample complexity guarantees as the former ones are any time guarantee.
The proposed algorithms are different since the goal is different.
In particular, the uniform concentration bound challenge does not appear there.
Note that using the explore-then-commit algorithm [23], one can achieve ˜ O
( T 2 / 3 ) regret for large T (from ˜ O
(1 /ϵ 2 ) sample complexity bound achieved in [24]) which is worse than ours.
Additionally, we achieve zero violation (Remark 2) while maintaining the same order of regret with respect to T .
4.2 Outline of the Proof
In this section, we provide an outline of our proof, which is mainly divided into three steps.
We first establish a decomposition of the sum of reward regret and constraint violation.
Then, we will bound two key terms that are related to optimism and prediction error, respectively.
Finally, using standard optimization tools, we can achieve the main results.
We highlight that the key challenges lie in the second step where a balance between the optimistic term and prediction error term is handled via the introduced soft-max policy.
Step 1:
Bounding the sum of Regret and violation scaled by dual variable Similar to [1], we first establish the following decomposition, which upper bounds the sum of regret and violation.
This will serve as the basis when applying optimization tools in Step 3.
Lemma 2 (Decomposition) .
For any Y ∈ [0 , ξ ] , we have
[Equation]
Note that T 1 is similar to the term related to optimism in the unconstrained case with the difference being that we now have two value functions weighted by the dual variable Y k .
Similarly, T 2 is similar to prediction error term with the additional weight by Y .
Since the first term in the above inequality can be easily bounded with a proper choice of η , we are only left to bound T 1 and T 2 , respectively.
Step 2: Bounding T 1 and T 2 To bound T 2 and T 1 , we need to bound the difference between the individual estimated value function V kj,h and the individual value function V πj,h corresponding to a given policy π at episode k .
As in the unconstrained case, the key step is to control the fluctuations in least-squares value iteration.
In particular, we need to show that for all ( k, h ) ∈
[ K ] ×
[ H ] with high probability
[Equation]
is upper bounded by lower order term (e.g., O ( d √ log K ) .
To this end, value-aware uniform concentration is required to handle the dependence between V kj,h +1 and samples { x τh +1 } k − 1 τ =1 , which renders the standard self-normalized inequality infeasible in the model-free setting.
The general idea here is to fix a function class V j,h in advance and then show that each possible value function in our algorithm V kj,h is within this class which has polynomial log-covering number.
In the following, we fix an h ∈ [ H ] and drop the subscript h for notation simplicity.
Uniform Concentration Bound for class of value function: We first note that this uniform concentration bound is the main motivation for us to choose a soft-max policy as we will see that the standard greedy policy would fail in this case.
That is, in order to guarantee that for each possible V kj , there is an ϵ -close function in V j , it would basically lead to a very large covering number.
To address this, we introduce soft-max policy and define the following corresponding function classes.
We first define the following class for Q -function for j = r, g .
[Equation]
( · , · ) T (Λ h ) − 1 ϕ ( · , · ) , H } .
Then, we define the following value function class V j .
V j
= { V j | V j ( · ) = (cid:80) a π ( a |· ) Q j ( · , a ); Q j ∈ Q j , π ∈ Π } , where Π is given by the following class Π = { π | π ( a |· ) =
S OFT -M AX aα (( Q r ( · , · ) + Y Q g ( · , · )); ∀ a ∈ A , Q r ∈
Q r , Q g ∈ Q g , Y ∈ [0 , ξ ] } , where S OFT -M AX is defined in (7).
Why soft-max?
At this moment, we can explain why the introduction of soft-max in our algorithm is critical.
Suppose we follow the standard greedy selection, which corresponds to α = ∞ in above.
The key issue in this approach is that one needs a large ϵ -covering for V j so that each possible V kj can be well-approximated (i.e., ϵ -close) by function in V j .
This is in sharp contrast to the unconstrained case where V j has a polynomial log-covering number.
To see this difference, in the unconstrained case, we only have V kr that is greedy with respect to Q kr .
By the fact that max a is a contraction map, an ϵ -covering of the Q kr implies an ϵ -covering of V kr and meanwhile the covering number of Q r is reasonably small (Lemma 14) by standard arguments.
This no longer holds in the constrained case due to the use of a composite Q -function.
In particular, note that if the policy is greedy w.r.t.
the composite Q -function, then an ϵ -covering of Q kj fails to be an ϵ -covering of V kj in general since the greedy policy is not smooth in that a slight change of the composite Q -function could lead to a substantial change of the output action.
This leads to a large distance for individual value functions due to the different action choices, even though the
Q -function is close (Please see Appendix G for an example).
Hence, one can not approximate individual value function within ϵ -bound using greedy policy based on composite Q -function.
This fact motivates us to turn to S OFT -M
AX α , which is Lipschitz continuous with a Lipschitz constant at most 2 α .
Thus, our main idea is as follows.
Given Q kr , Q kg and Y k , we can first find fixed ˜ Q r ∈ Q r , ˜ Q g ∈ Q g and ˜ Y ∈ [0 , ξ ] such that (cid:13)(cid:13)(cid:13) Q kr − ˜ Q
r (cid:13)(cid:13)(cid:13) ∞
≤ ϵ 1 , (cid:13)(cid:13)(cid:13) Q kg − ˜ Q g (cid:13)(cid:13)(cid:13) ∞ ≤ ϵ 2 , (cid:13)(cid:13)(cid:13)
Y k − ˜ Y
(cid:13)(cid:13)(cid:13) ∞ ≤ ϵ 3 and (cid:13)(cid:13)(cid:13) ( Q kr + Y k Q kg ) − ( ˜ Q r + ˜ Y ˜ Q g ) (cid:13)(cid:13)(cid:13) ∞ ≤ ϵ with a reasonably small covering number.
Then, thanks to the smoothness of soft-max function, we have ∥ π k − ˜ π
∥ 1 ≤ 2 αϵ (Lemma 15).
Combining this with the closeness of individual Q -function yields the closeness of individual V -function (Lemma 13).
Hence, it ensures that the class V j in our set-up has log-covering number of O ( d log( K )) .
Choosing hyper-parameter α to achieve bound: A larger value of α means that we need a smaller ϵ , hence a larger covering number.
Then, one may wonder if we can choose an arbitrarily small value for α .
However, the term T 1 will be enlarged if we choose too small α .
Note that in the unconstrained case, T 1 is upper bounded by zero due to optimism under greedy policy.
Now, since we are using soft-max, we need to bound the approximation error between the soft-max and greedy one.
As expected, in this case, a larger α leads to a smaller approximation error.
From the discussions above, we can see that the approximation-smoothness trade-off of the soft-max function is well captured by our T 1 and T 2 , respectively.
Therefore, we need to carefully choose the value of α to balance these two.
In particular, with α = log( |A| ) K 2 H , we have the following bounds on
T 1 and T 2 , respectively.
Lemma 3.
With probability 1 − p/ 2 , we have T 1 ≤ K H log( |A| ) α .
Hence, for α = log( |A| ) K 2(1 + ξ + H ) , we have T 1 ≤ 2 H (1 + ξ + H ) with probability 1 − p/ 2 .
Lemma 4.
With probability at least 1 − p/ 2 , T 2 ≤ O (( Y + 1) √ d 3 H 3 Tι 2 ) , where ι = log[log( |A| )4 dT/p ]
Remark 1.
The additional log | A | factor in ι arises as a trade-off for selecting soft-max policy as it is evident in Lemma 3.
When we compensate by making α scaled with log | A | in Lemma 3, it increases the covering-number by log | A | as well in the ι term.
Step 3: Final Result by combining all the pieces: By replacing Y = 0 , η = ξ/ ( √ KH 2 ) , and combining Lemma 2,3, and 4, we obtain the regret bound.
We also obtain the constraint violation using the idea from [1].
Remark 2.
We can reduce the violation to zero while maintaining the same order on regret with respect to T (Appendix H).
We consider a tighter optimization problem where we add ζ in the constraint of (4).
In Appendix H, we bound the difference between the optimal value function for the tighter and the original problem as a function of ζ .
Since the tighter problem is also CMDP, we attain the regret and violation bound as in Theorem 1 with b + ζ in place of b .
Hence, by choosing ζ , we can show that it is possible to achieve ˜ O ( √ T ) regret and zero violation for large enough K (Theorem 3 in Appendix H) albeit with an extra H factor in front of regret bound.
Experiments
We evaluate Algorithm 1 on a simulated model for job scheduling to validate our theoretical results.
We consider that the number of jobs belongs to the discrete state { 0 , 1 , . . .
, 9 } where 0 means that there is no job.
Total time horizon ( H ) is divided in 10 steps.
After H steps, a new episode begins.
At the start of the each episode, the state of the job is assumed to be 9 , i.e., the job stack is full.
The agent needs to decide whether to send job ( a = 1) or not ( a = 0 ) to a machine.
We assume that if a = 1 , the agent sends 2 jobs to a machine and incurs a cost as the machine spends some resources to process the job.
We assume that the rewards are step dependent.
In particular, we assume that at time steps from 3 to 6 , the reward is 1 − 0 .
9 a , In other time steps, the reward is 1 − 0 .
2 a .
This mimics the setup where at certain time, it might be more costly to process a job (for example, electricity cost might be higher, or the machine needs to abandon an important job).
[Figure]
t r .
V i
o l a t i on
We assume that even if the scheduler schedules a job, the machine might not be able to complete the 2 jobs.
In particular,
[Equation]
Thus, if a = 0 , the state x h +
= x h .
The agent gets an utility of g ( x h , a h , x h +1 )
=
( x h − x h +1 ) / 2 .
We want that utility to be greater than or equal to 4 at the end of every episode.
This will ensure that at most 1 job can remain at the end of each episode.
We run Algorithm 1 for 2 × 10 5 episodes ( K ).
The parameters we used are the followings: α =
K/ (1 + 2 H/γ + H ) , η = 2 H/ ( γ √ KH 2 ) .
We set γ = 1 .
We have also set ϵ = 0 .
1 in order to ensure that the violation goes towards 0 as K increases.
Note that the setup can be represented in a tabular form.
Since linear MDP contains tabular form, the feature space representation becomes simple, in particular, ϕ ( s, a )
=
e s,a where e s,a is 1 for the state-action pair ( s, a ) , and 0 otherwise.
The dimension of the feature space is |S||A| .
The cumulative regret and constraint violations are shown in Figure 1.
As predicted by our theory, the regret scales only as √ K .
On the other hand, the cumulative violation oscillates.
However, the violation approaches 0 as K increases.
The oscillation of violation is due to the fact that dual variable also oscillates in order to illicit conservative response when the cumulative violation increases, and illicit aggressive response when the violation decreases.
Conclusion and Future Work
We propose a model-free RL-based algorithm for linear MDP.
We have achieved ˜ O ( √ d 3 H 3 T ) regret and ˜ O ( √ d 3 H 3 T ) constraint violation.
To the best of our knowledge, this is the first result which shows ˜ O ( √ T ) regret and ˜ O ( √ T ) constraint violation without requiring a generator for the model-free case.
We have extended the LSVI-UCB algorithm in the primal-dual type framework.
We have underlined the technical challenges in doing so and explained how the greedy policy fails to achieve an uniform concentration bound for individual value function.
Subsequently, we show that a soft-max type algorithm achieves that.
Compared to [7], our regret is off by √ d factor which is due to the fact that we need to use uniform concentration bound.
The similar gap is also observed for the unconstrained set-up as well.
Whether we can tighten this dependence on d remains an important future research direction.
Whether we can tighten the dependence on H also constitutes a future research direction.
Extending the work to the setup where the feature-space needs to be learnt is also important.
Recent works [25–27] on feature-space learning for unconstrained MDPs may provide some insights in this direction.
Consideration of non-linear MDP also constitutes a future research direction.
Acknowledgment
XZ is partially supported by the NSF grant CNS-2153220.
This work has been supported in part by
NSF grants: 2112471 (also partly funded by DHS), CNS-2106933, and CNS-1901057 and a grant from the Army Research Office: W911NF-21-1-0244.