Provably Efficient Model-Free Constrained RL with Linear Function Approximation
Anonymous Author(s)
AffiliationAddress email
Abstract
We study the constrained reinforcement learning problem, in which an agentaims to maximize the expected cumulative reward subject to a constraint on theexpected total value of a utility function.
In contrast to existing model-basedapproaches or model-free methods accompanied with a ‘simulator’, we aim todevelop the first model-free , simulator-free algorithm that achieves a sublinearregret and a sublinear constraint violation even in large-scale systems.
To thisend, we consider the episodic constrained Markov decision processes with linearfunction approximation, where the transition dynamics and the reward function canbe represented as a linear function of some known feature mapping.
We show that˜ O ( √ d 3 H 3 T ) regret and ˜ O ( √ d 3 H 3 T ) constraint violation bounds can be achieved,where d is the dimension of the feature mapping, H is the length of the episode, and
T is the total number of steps.
Our bounds are attained without explicitly estimatingthe unknown transition model or requiring a simulator, and they depend on thestate space only through the dimension of the feature mapping.
Hence our boundshold even when the number of states goes to infinity.
Our main results are achievedvia novel adaptations of the standard LSVI-UCB algorithms.
In particular, wefirst introduce primal-dual optimization into the LSVI-UCB algorithm to balancebetween regret and constraint violation.
More importantly, we replace the standardgreedy selection with respect to the state-action function in LSVI-UCB with asoft-max policy.
This turns out to be key in establishing uniform concentration forthe constrained case via its approximation-smoothness trade-off.
Finally, we alsoshow that one can achieve an even zero constraint violation for large enough T bytrading the regret a little bit but still maintaining the same order with respect to T .
Introduction
In many practical applications of online reinforcement learning (RL) (e g., financial regulations,safety), there exist additional constraints on the learned policy in the sense that it also needs to ensurethat the expected total utility (cost, resp.) exceeds a given threshold (is below a threshold, resp.).
Constrained RL problem is formulated as a constrained Markov Decision Process (CMDP), in whichthe celebrated exploration-exploitation trade-off in online RL becomes more challenging due to theadditional need to find a balance between reward regret and constraint violations.
To develop online sample-efficient algorithms for CMDPs, prior works have largely resorted tomodel-based approaches, where the control policy is constructed based on a learned model [1–7].
However, due to the explicit estimation and storage of the unknown transition model, model-basedapproaches often lead to large time and space complexities.
These issues are exacerbated in the largestate space.
Hence, there are several recent works starting to investigate model-free algorithms for
CMDPs, which directly update the value function or the policy without first estimating the model [8–10].
However, all of these works consider an easier setting compared to standard RL in that they assume access to a simulator [11] (a.k.a.
a generative model [12]), which is a strong oracle that allowsthe agent to query arbitrary state-action pairs and return the reward and the next state, hence greatlyalleviating the intrinsic difficulty of exploration in RL.
To the best of our knowledge, [13] were thefirst to study model-free and simulator-free algorithms for CMDPs.
The above mentioned work onmodel-free algorithm considers the finite-state tabular setting and the regret scales polynomially withthe number of states.
Thus, the result would not be useful for large-scale RL applications where thenumber of states could even be infinite.
To address this curse of dimensionality, modern RL hasadopted function approximation techniques to approximate the (action-)value function or a policy,which greatly expands the potential reach of RL, especially via deep neural networks.
However,little is known for the performance guarantee of model-free algorithms in CMDPs beyond tabularsettings, even in the case of linear function approximation.
Motivated by this, we are interested in thefollowing question:
Can we achieve provably sample-efficient and model-free exploration for CMDPs beyond tabularsettings (without a simulator)?
Contribution.
To answer the above question, we consider the episodic CMDPs with linear functionapproximation, where the transition dynamics and the reward function can be represented as a linearfunction of some known feature mapping.
Our main contributions are as follows.
• We show that with a proper parameter choice, our proposed algorithm achieves ˜ O ( √ d 3 H 3 T ) regretand ˜ O ( √ d 3 H 3 T ) constraint violation bounds with a high probability, where d is the dimension of
the feature mapping, H is the length of the episode, and T is the total number of steps.
We also
show that it is in fact possible to achieve zero constraint violation by trading the regret a little bit
while maintaining the same order with respect to T .
• Our bounds are attained without explicitly estimating the unknown transition model or requiring a
simulator , and they depend on the state space only through the dimension of the feature mapping.
To the best of knowledge, these sub-linear bounds are the first results for model-free, simulator-freeonline RL algorithms for CMDPs with function approximations.
We even improve the bound( ˜ O ( T 0 . 8 ) ) proposed in the model-free finite state tabular setting by [13] (Table 1).• We combine the primal-dual algorithm with the classic model-free, simulator-free LSVI-UCBalgorithm [14] to balance between regret and constraint violations.
This naturally leads to the
construction of a new composite state-action function (i
e., Q -function), which is the sum of theQ -function for the reward and the Q -function for the utility weighted by the dual variable.
Due to
this new type of Q -function in CMDPs, a key challenge arises when establishing the value-aware
uniform concentration, which lies at the heart of the performance analysis of model-free exploration.
More specifically, the standard greedy selection with respect to this composite Q -function failsin finding non-trivial covering number for the function class of individual value functions (i.e.,
V -function) for the reward and the utility respectively.
To address this fundamental issue, weinstead adopt a soft-max policy by utilizing its nice property of approximation-smoothness trade-offvia its parameter, i.e., temperature coefficient.
1.1 Related Work
Model-based RL algorithms have been proposed for the CMDP [1–3, 5–7].
Apart from [7], the restconsidered tabular set-up.
In the tabular model-based set-up, the best known regret and constraintviolations achieved are ˜ O
( (cid:112) |S| 2 |A| T ) where |S| and |A| are the dimensions of the state and actionspaces respectively.
Hence, such results can not cope up with the large state space.
[7] consideredlinear kernel MDP whereas we consider linear MDP.
These two are not the same in general.
Wedescribe the differences with the algorithm proposed in [7] in Section 4.1.
Model-free RL algorithms have also been proposed [8–10] to solve CMDP.
However, all of theabove require a generator model, which simulates from any state and action.
Further, their proposedalgorithms bound the expected regret but does not guarantee a regret with high probability.
[13]proposed a ‘triple-Q’ algorithm which does not require a ’simulator’.
However, it only consideredtabular setting.
The regret bound shown in the above paper is ˜ O
( T 0 . 8 ) which is far from the optimalfor the model-based case ˜ O ( √ T ) .
Please see Table 1 to see our contribution compared to the stateof-the-art approaches.
[15] proposed a RL algorithm for the scenario where a constraint needs tobe satisfied at each step of an episode.
We consider a constraint where the cumulative utility over
[Table]
the length of the episodes must exceed a threshold.
Hence, the set of constraints is fundamentallydifferent.
The authors in [15] also assumed that a safe-action is known for each state which we donot assume in our setting
[Table]
Problem Formulation
We consider an episodic constrained MDP, denoted by ( S , A , P , H, r, g ) where S is the state space,
A is the action space, H is the fixed length of each episode, P = { P h } Hh =1 is a collection oftransition probability measures,
r = { r h } Hh =1 is a collection of reward functions, and g = { g h } Hh =1is
a collection of utility functions.
We assume that S is a measurable space with possibly infinitenumber of elements, A is a finite action set.
P h ( ·| x, a ) is the transition probability kernel whichdenotes the probability to reach a state when action a is taken at state x .
r h : S ×
A → [0 , 1] , andg h : S
×
A → [0 , 1] and are assumed to be deterministic.
However, we can readily extend to settingswhen r h and g h are random.
Each episode k ∈ [ K ] starts with the fixed state x 1 .
It can be readily generalized to the settingwhere x 1 is drawn from a distribution.
Then at each step h ∈ [ H ] in episode k , the agent observesstate x kh ∈ S , picks an action a kh ∈ A , receives a reward r h ( x kh , a kh ) , and a utility g h ( x kh , a kh ) .
The
MDP evolves to x kh +1 that is drawn from P h ( ·| x kh , a kh ) .
The episode terminates at step H + 1 .
Without loss of generality, we assume that r H +1 =
g H
+
= 0 .
In this paper, we consider thechallenging scenario where the agent only observes the bandit information r h ( x kh , a kh ) and g h ( x kh , a kh )at the visited state-action pair ( x kh , a kh ) .
The policy-space of an agent is ∆( A|S , H ) ; {{ π h ( ·|· ) } Hh =1 :π h ( ·| x ) ∈ ∆( A ) , ∀ x ∈ S , and h ∈
[ H ] }.
Here ∆( A ) is the probability simplex over the action space.
For any x kh ∈ S , k ∈ [ K ] , and h ∈
[ H ] , π h,k ( a kh | x kh ) denotes the probability that the action a kh ∈ Ais taken at episode k when the state is x kh .
Let V πr,h ( x ) denote the expected value of the total reward function starting from step h and state xwhen the agent selects action using the policy π = { π h }
[Equation] [Equation]
where E is taken with respect to the policy π and the transition probability kernel P .
Let Q πr,h ( x, a )denote the expected value of the total reward starting from step h and the state-action pair ( x, a ) andfollows the policy π as
[Equation]
Similarly, we define the value function for the utility V πg,h ( x ) , and the action-value function for theutility Q πg,h ( x, a ) .
We denote V πj,h ( x ) , and Q πj,h ( x, a ) for j = r, g .
Definition 1.
For brevity, we denote P h V πj,
h +1 ( x, a ) =
E x ′ ∼ P h ( ·| x,a ) V πj,
h +1 ( x ′ ) for j = r, g .
Using this notation, the Bellman’s equation associated with the policy π becomes
[Equation]
Note that
[Equation]
where
[Equation]
The objective of the learning agent is to find an optimal solution of the following problem
[Equation]
subject
[Equation]
Note that even though we have only once constraint, it can be readily generalized to the scenariowith multiple constraints.
In order to avoid trivial solutions, we consider b ∈ (0 , H ] .
We denote theoptimal policy as π ∗ which solves the above optimization problem.
Since π ∗ is obtained by havingcomplete information, it is also denoted as the best policy in the hindsight .
Without any constraint information a priori, an agent can not know the policies that satisfy theconstraint.
Instead, we allow the policy to violate the constraint and minimize the regret whileminimizing the total constraint violations over the K episodes.
Such an approach is also consideredin the existing literature [1, 7, 9].
We now define the performance metric which we seek to minimize.
Performance Metric.
Let the policy employed by the agent at episode k be π k =[ π 1 ,k , . .
, π h,k , . . .
, π H,k ] T .
The performance metric we are considering is the following
[Equation]
where [ z ] + = max { z, 0 } .
The regret is defined as the difference between the total reward value byfollowing the optimal policy π ∗ , and the total reward value obtained by following agent’s policy π k atepisode k over K episodes.
The constraint violation is defined as the difference between the thresholdvalue Kb and the total utility function attained by following the policies over all the episodes K .
Linear Function Approximation.
To handle a possible large number of states, we consider thefollowing linear MDPs.
Assumption 1.
The CMDP is a linear MDP with feature map ϕ : S ×
A → R d , if for any h,there exists d unknown signed measures µ h = { µ 1 h , . .
. , µ dh } over S such that for any ( x, a, x ′ ) ∈
S ×
A × S ,
[Equation]
and there exists vectors θ r,h , θ g,h ∈ R d such that for any ( x, a ) ∈ S × A ,
[Equation]
Assumption 1 adapts the definition of linear MDP [14, 16] to the constrained case.
By the abovedefinition, the transition model, the reward, and the utility functions are linear in terms of feature mapϕ .
We remark that despite being linear, P h ( ·| x, a ) can still have infinite degrees of freedom
sinceµ h ( · ) is unknown.
Note that tabular MDP is part of linear MDP [14].
Note that [7, 17] studied another related concept known as linear kernel MDP.
In the linear kernel
MDP, the transition probability is given by P h ( x ′ | x, a ) = ⟨ ψ ( x ′ , x, a ) , θ h ⟩ .
In general, linear MDPand linear kernel MDPs are two different classes of MDP [17].
Similar to Proposition 1 in [14], we can show that for a linear MDP and for any policy π there exists{ w πj,h } Hh =1 such that Q πj,h ( x, a ) = ⟨ w πj,h , ϕ
( x, a ) ⟩ for any ( x, a, h ) ∈ S
×
A ×
[ H ] .
We, thus, focuson linear action-value function.
Dual problem and Slater’s Condition.
We first introduce few notations which we will use throughoutthis paper.
Definition 2.
V π,Yh ( · ) =
V πh,r ( · ) + Y V πh,g ( · ) , and Q π,Yh ( x, a ) =
Q πr,h ( x, a ) + Y Q πg,h ( x, a ) , where
Y is the dual variable.
Thus, V π,Yh ( · ) and Q π,Yh ( · , · ) are respectively the composite value function and Q -functions respectively.
We can cast the problem (4) as a saddle point problem max π min Y L ( π, Y ) where
L ( π, Y ) =
V πr, 1 ( x 1 ) + Y ( V πg, 1 ( x 1 ) − b ) =
V π,Y− Y b , where π is the primal policy and Y is thedual variable.
However, the lagrangian is non-concave in π [18] even though it is convex in Y .
Nevertheless, the strong duality holds [19].
Hence, there exists optimal dual variable Y ∗ , such thatmax π L ( π, Y ∗ ) will correspond to the optimal reward value function.
We assume the following slater’s condition in this paper.
Assumption 2 (Slater’s Condition) .
There exists γ > 0 , and ¯ π ∈ ∆( A|D , H ) , such that V ¯ πg, 1 ( x 1 ) ≥b + γ ,
Lemma 1 (Boundedness of Y ∗ ) .
The optimal dual-variable Y ∗ ≤ V π [Equation] [Equation] −
[Equation] [Equation] The slater’s condition is mild in practice and commonly adopted in previous works [7, 1, 20].
We usethe properties of the slater’s condition to bound the performance of our proposed algorithm.
Definition 3.
We set ξ = 2 H/γ .
Our Approach
1: Initialization:
[Equation]
p ) 2: for episodes k
, K do do
: Λ
← (cid:80) k − 1 τ =1 ϕ ( x τh , a τh ) ϕ
( x τh , a τh ) T
+ λ I 6: w kr,h ← (Λ kh ) − 1 [ (cid:80)
k − 1 τ =1 ϕ
( x τh , a τh )
[ r h ( x τh , a τh ) + V kr,h +
( x τh +1 )]] 7: w kg,h ← (Λ kh ) − 1 [ (cid:80)
k − 1 τ =1 ϕ
( x τh , a τh )
[ g h ( x τh , a τh ) + V kg,h +
( x τh +1 )]] 8:
Q kr,h ( · , · ) ← min {⟨ w kr,h , ϕ ( · , · ) ⟩ + β ( ϕ ( · , · ) T (Λ kh ) − 1 ϕ ( · , · )) 1 / 2 , H } 9:
Q kg,h ( · , · ) ← min {⟨ w kg,h , ϕ ( · , · ) ⟩ + β ( ϕ ( · , · ) T (Λ kh ) − 1 ϕ ( · , · )) 1 / 2 , H } 10: π h,k ( a |· ) = exp( α ( Q kr,h ( · , a ) + Y k Q kg,h ( · , a )))
(cid:80) a exp( α ( Q kr,h ( · , a ) + Y k Q kg,h ( · , a ))) 11:
V kr,h ( · ) =
(cid:80) a π h,k ( a |· ) Q kr,h ( · , a ) 12:
V kg,h ( · ) =
(cid:80) a π h,k ( a |· ) Q kg,h ( · , a ) 13: for step h = 1 , . . .
, H do 14:
Compute Q kr,h ( x kh , a ) , Q kg,h ( x kh , a ) , π ( a | x kh ) for all a .
Take action a kh ∼ π h,k ( ·| x kh ) and observe x kh +1 .
[Equation]
We now describe our proposed algorithm in Algorithm 1.
This algorithm is based on the primal-dualadaptation of the LSVI-UCB [14].
For a given dual variable, the primal policy is updated, and thenthe dual value is updated based on the estimated utility value function.
At each episode, the algorithmconsists of three parts.
The first part (Steps 4-12) consists of updating the parameters w kr,h , w kg,h andΛ kh which are used to update the Q kj,h and V kj,h at episode k .
Note that the Steps 8-12 are not evaluatedfor each state, rather, they are evaluated only for the encountered states till episode k − 1 .
Hence, wedo not need to iterate over potentially infinite number of states.
For the first episode, since k − 1 = 0and τ = 1 , we have w kj,h = 0 , ∀ j and Λ kh
= λ
I .
We note that Q kj,H +1 ( · , · ) = 0 for j = r, g .
The value functions are updated (Steps 11-12) based on Q function and the policy.
The policy isbased (Step 10) on a soft-max policy unlike the greedy one in the unconstrained case [14].
Soft-maxpolicy S OFT -M
AX α
( X ) =
{ S OFT -M
AX iα ( X ) } |A| i =1 for any vector X ∈ R |A| is a |A| -dimensionalvector with parameter α where the i -th component
[Equation]
At step h , π h,k ( a | x ) is computed based on the soft-max policy on the composite Q -function vector{ Q kr,h ( x, a ) + Y k Q kg,h ( x, a ) } a ∈A where Y k is the lagrangian multiplier.
When α = ∞ , this becomesequal to the greedy policy.
The second part (Steps 13-15) is the execution of the soft-max policybased on the composite Q -value for the encountered state x kh .
Q function and Value function Estimation.
We need to estimate the value-function and Q -functionwith respect to the policy π k .
However, there are challenges.
We do not know P h in Bellman’sequation (3), rather P h
V
π k j,h +1 should be replaced by the empirical samples.
Further, in the largestate space, we can not iterate over all ( x, a ) .
Rather, we parameterize Q π ∗ j,h ( · , · ) by a linear form⟨ w kj,h ( · , · ) , ϕ ( · , · ) ⟩ .
The intuition is to obtain w kj,h from the Bellman’s equation using the regularizedleast-square regression.
We obtain w kj,h for j = r, g according to the following equation
[Equation]
Then, an additional bonus term β ( ϕ ( · , · ) T (Λ kh ) − 1 ϕ ( · , · )) 1 / 2 is added as in [14].
β is constant whichwe will characterize in the next section.
Λ kh is the Gram matrix for the regularized least squareproblem.
Such an additional term is used for upper confidence bound in LSVI-UCB [14].
The sameadditional term is used for both Q kr,h and Q kg,h .
Note the difference with the LSVI-UCB, here, weneed to estimate the value function corresponding to the soft-max policy π k where in LSVI-UCB, agreedy policy corresponding to the Q -function is used.
Policy.
We update a soft-max policy which selects actions according to the estimated ‘composite’
Q -function at the k -th episode.
The reason behind using the soft-max policy instead of a greedy policywill be apparent in the next section when we state the main results and the proof ideas.
Note fromthe strong duality, for optimal dual variable Y ∗ , optimal primal policy π ∗ maximizes the compositevalue function V π ∗ ,Y ∗ (Definition 2).
Thus, the optimal policy should be a greedy one based onthis optimal dual value Y ∗ .
However, the greedy policy is not Lipschitz, hence, it does not provideuniform concentration bound for each individual value function, an essential step in the regret bound(Section 4.2).
Hence, compared to the unconstrained scenario, we need more exploration in the policyspace where the apparent reason is that we do not know the optimal dual variable beforehand.
Sincewe use the soft-max policy, there is a gap compared to the optimal value even when the lagrangianmultiplier Y k becomes equal to Y ∗ .
However, if α in the soft-max policy also scales with K , then wecan bound the gap from the optimal value function (Section 4.2).
Dual Update.
To infer the constraint violation, we estimate V k g, 1 for V π k g, 1 .
We update the lagrangianmultiplier Y k by moving towards minimizing the lagrangian L ( π, Y ) over Y ≥ 0 in line 16, whereη > 0 is a step-size and ξ is the upper bound on the dual variable such that optimal dual variable Y ∗is contained within [0 , ξ ] .
The dual update is similar to the step described in [7, 1].
The dual update works as a trade-off between the reward maximization and the constraint violationreduction.
If b − V kg, 1 ≥ 0 , that means with a high probability, the constraint will be violated for thepolicy π k .
Hence, the dual value is increased in order to focus on minimizing the constraint violation.
Otherwise, the agent tries to maximize the reward value function.
Space and Time Complexities.
We remark that Algorithm 1 only needs to store Y k ,w kr,h , w kg,h , r h ( x kh , a kh ) , g h ( x kh , a kh ) , Λ kh , and { ϕ ( x kh , a ) } a ∈A for
all ( h, k ) ∈
[ H ] ×
[ K ] , hence, ittakes O ( d 2 H + d A T ) space.
When we compute (Λ kh ) − 1 using Sherman-Morrison formula, the computation of V kj,h +1 is dominated by computing Q kj,h +1 and the policy π k .
Hence, it takes O ( d 2 A T )time.
Note that since our approach is model-free and we do not need to evaluate integrals as in [7] inorder to estimate P
h V
kh +1 .
Note that both η and α use the knowledge of K .
In case, K is unknown, one can use the “doublingtrick" [21] which will only scale the regret and constraint violation by a constant factor.
Analysis
We now state the main result.
We prove that Algorithm 1 achieves regret and constraint violationwhich are sublinear in T =
KH where T is the total number of steps.
4.1 Main Results
Theorem 1.
Fix p > 0 .
If we set λ = 1 , β = C 1 dH √ ι in Algorithm 1 where ι = log(log( |A| )4 dT/p )for some absolute constant C 1 .
With probability (1 − p ) ,
[Equation]
for some absolute constants C , and C ′ .
We remark the difference with the existing results.
Since ξ = 2 H/γ (by Definition 3), our resultindicates that our approach obtains ˜ O ( √ d 3 H 3 T ) regret and the same order of constraint violationwhere ˜ O absorbs logarithmic factor on T .
The regret and constraint violation are sub-linear in T ,and similar dependence is observed in [1, 7].
Also note that compared to the unconstrained case[14], there is an additional log( |A| )) factor in the value of ι which arises because we use soft-maxpolicy instead of the greedy policy which adds to the covering number.
The regret and constraintviolation do not depend on the dimension of the state space, rather, it depends on the dimension ofthe feature space.
To the best of our knowledge this is the first result which shows both ˜ O ( √ T ) regretand constraint violation in the model-free set up (tabular or linear) without requiring a simulator.
Comparison with [7] : Compared to [7] which also considers linear function approximation (however,it considers linear kernel MDP rather linear MDP)
we improve the result in [7] by a factor of H .
Second, compared to [7], which is a model-based policy-based algorithm, ours is a model-freevalue-based algorithm.
Due to this, the above uniform concentration challenge does not exist in [7].
Moreover, our model-free algorithm also enjoys an easy implementation and improved computationefficiency since it does not estimate the next step expected value function as in [7] which requires anintegration oracle to compute a d -dimensional integration at every step.
[7] also needs to store theprevious policies and estimated value functions, hence, it needs O ( T ) additional space complexity.
Hence, our algorithm is conducive for implementation.
We have an additional √ d factor in front of the regret and constraint violation.
Similar differencein regret is also observed between the model-based linear kernel unconstrained MDP [22] andmodel-free linear unconstrained MDP [14] even in the unconstrained case .
Similar to the discussion in Section 3.1 on [23], our result directly translates to a sample complexityguarantee (or, PAC guarantee).
For example, we can learn a policy π such that V π ∗ r, 1 ( x 1 ) − V π r, 1 ( x 1 ) ≤ϵ , and b − V πg, 1 ( x 1 ) ≤ ϵ after ˜ O
( d 3 H 4 /ϵ 2 ) number of samples.
Here, the policy π is obtainedafter running Algorithm 1 for ˜ O
( d 3 H 3 /ϵ 2 ) number of episodes, and then selecting policy π k withprobability 1 /K for any k ∈
[ K ] .
4.2 Outline of the Proof
In this section, we provide an outline of our proof, which is mainly divided into three steps.
We firstestablish a decomposition of the sum of reward regret and constraint violation.
Then, we will boundtwo key terms that are related to optimism and prediction error, respectively.
Finally, using standardoptimization tools, we can achieve the main results.
We highlight that the key challenges lie in thesecond step where a balance between the optimistic term and prediction error term is handled via theintroduced soft-max policy.
Step 1.
Similar to [1], we first establish the following decomposition, which upper bounds the sum ofregret and violation.
This will serve as the basis when applying optimization tools in Step 3.
Lemma 2 (Decomposition) .
For any Y ∈ [0 , ξ ] , we have
[Equation]
Note that T 1 is similar to the term related to optimism in the unconstrained case with the differencebeing that we now have two value functions weighted by the dual variable Y k .
Similarly, T 2 is similarto prediction error term with the additional weight by Y .
Since the first term in the above inequalitycan be easily bounded with a proper choice of η , we are only left to bound T 1 and T 2 , respectively.
Step 2. To bound T 2 and T 1 , we need to bound the difference between the individual estimated valuefunction V kj,h and the individual value function V πj,h corresponding to a given policy π at episode k .
As in the unconstrained case, the key step is to control the fluctuations in least-squares value iteration.
In particular, we need to show that for all ( k, h ) ∈
[ K ] ×
[ H ] with high probability
[Table]
is upper bounded by lower order term (e.g., O ( d √ log K ) .
To this end, value-aware uniform concentration is required to handle the dependence between V kj,h +1 and samples { x τh +1 } k − 1 τ =1 , whichrenders the standard self-normalized inequality infeasible in the model-free setting.
The general ideahere is to fix a function class V j,h in advance and then show that each possible value function in ouralgorithm V kj,h is within this class which has polynomial log-covering number.
In the following, wefix an h ∈
[ H ] and drop the subscript h for notation simplicity.
We first note that this uniform concentration bound is the main motivation for us to choose a softmax policy as we will see that the standard greedy policy would fail in this case.
That is, in orderto guarantee that for each possible V kj , there is an ϵ -close function in V j , it would basically leadto a very large covering number.
To address this, we introduce soft-max policy and define thefollowing corresponding function classes.
We first define the following class for Q -function forj = r, g .
Q j = { Q j | Q j ( · , · ) = min {⟨ w j , ϕ ( · , · ) ⟩ + β (cid:112) ϕ ( · , · ) T (Λ h ) − 1 ϕ ( · , · ) , H } .
Then, we definethe following value function class V j .
V j
= { V j |
[Equation] = [Equation] a π ( a |· ) Q j ( · , a ); Q j ∈ Q j , π ∈ Π } ,where Π is given by the following class Π = { π | π ( a |· ) =
S OFT -M AX aα (( Q r ( · , · )+ Y Q g ( · , · )); ∀ a ∈
A , Q r ∈ Q r , Q g ∈ Q g , Y ∈ [0 , ξ ] } , where S OFT -M AX is defined in (7).
At this moment, we can explain why the introduction of soft-max in our algorithm is critical.
Supposewe follow the standard greedy selection, which corresponds to α = ∞ in above.
The key issue in thisapproach is that one needs a large ϵ -covering for V j so that each possible V kj can be well-approximated(i.e., ϵ -close) by function in V j .
This is in sharp contrast to the unconstrained case where V j has apolynomial log-covering number.
To see this difference, in the unconstrained case, we only have V kr that is greedy with respect to Q kr .
By the fact that max a is a contraction map, an ϵ -covering ofthe Q kr implies an ϵ -covering of V kr and meanwhile the covering number of Q r is reasonably small(Lemma 14) by standard arguments.
This no longer holds in the constrained case due to the use of acomposite Q -function.
In particular, note that if the policy is greedy w.r.t.
the composite Q -function,then an ϵ -covering of Q kj fails to be an ϵ -covering of V kj in general since the greedy policy is notsmooth in that a slight change of the composite Q -function could lead to a substantial change of theoutput action.
This leads to a large distance for individual value functions due to the different actionchoices, even though the Q -function is close (Please see Appendix G for an example).
Hence, one cannot approximate individual value function within ϵ -bound using greedy policy based on composite Q -function.
This fact motivates us to turn to S OFT -M
AX α , which is Lipschitz continuous with a Lipschitz constant at most 2 α .
Thus, our main idea is as follows.
Given Q kr , Q kg and Y k , we can firstfind fixed ˜ Q r ∈ Q r , ˜ Q g ∈ Q g and ˜ Y ∈ [0 , ξ ] such that (cid:13)(cid:13)(cid:13) Q kr − ˜ Q
r (cid:13)(cid:13)(cid:13) ∞
≤ ϵ 1 , (cid:13)(cid:13)(cid:13) Q kg − ˜ Q g (cid:13)(cid:13)(cid:13) ∞ ≤ ϵ 2 ,(cid:13)(cid:13)(cid:13)
Y k − ˜ Y
(cid:13)(cid:13)(cid:13) ∞ ≤ ϵ 3 and (cid:13)(cid:13)(cid:13) ( Q kr + Y k Q kg ) − ( ˜ Q r + ˜ Y ˜ Q g ) (cid:13)(cid:13)(cid:13) ∞ ≤ ϵ with a reasonably small coveringnumber.
Then, thanks to the smoothness of soft-max function, we have ∥ π k − ˜ π
∥ 1 ≤ 2 αϵ (Lemma 15).
Combining this with the closeness of individual Q -function yields the closeness of individual Vfunction (Lemma 13).
Hence, it ensures that the class V j in our set-up has log-covering number of O ( d log( K )) .
In fact, a larger value of α means that we need a smaller ϵ , hence a larger covering number.
Then,one may wonder if we can choose an arbitrarily small value for α .
However, the term T 1 will beenlarged if we choose too small α .
Note that in the unconstrained case, T 1 is upper bounded byzero due to optimism under greedy policy.
Now, since we are using soft-max, we need to bound theapproximation error between the soft-max and greedy one.
As expected, in this case, a larger α leadsto a smaller approximation error.
From the discussions above, we can see that the approximation-smoothness trade-off of the soft-maxfunction is well captured by our T 1 and T 2 , respectively.
Therefore, we need to carefully choose thevalue of α to balance these two.
In particular, with α = log( |A| ) K 2(1+ ξ + H ) , we have the following bounds on
T 1 and T 2 , respectively.
Lemma 3.
With probability 1 − p/ 2 , we have T 1 ≤ K H log( |A| ) α .
Hence, for α = log( |A| ) K 2(1 + ξ + H ) ,we have T 1 ≤ 2 H (1 + ξ + H ) with probability 1 − p/ 2 .
Lemma 4.
With probability at least 1 − p/ 2 , T 2 ≤ O (( Y + 1) √ d 3 H 3 Tι 2 ) , where ι =log[log( |A| )4 dT/p ]
Remark 1.
The additional log | A | factor in ι arises as a trade-off for selecting soft-max policy as it isevident in Lemma 3.
When we compensate by making α scaled with log | A | in Lemma 3, it increasesthe covering-number by log | A | as well in the ι term.
Step 3 : By replacing Y = 0 , η = ξ/ ( √ KH 2 ) , and combining Lemma 2,3, and 4, we obtain theregret bound.
We also obtain the constraint violation using the idea of [1].
Remark 2.
We can reduce the violation to zero while maintaining the same order on regret withrespect to T (Appendix H).
We consider a tighter optimization problem where we add ζ in theconstraint of (4).
In Appendix H, we bound the difference between the optimal value function for thetighter and the original problem as a function of ζ .
Since the tighter problem is also CMDP, we attainthe regret and violation bound as in Theorem 1 with b + ζ in place of b .
Hence, by choosing ζ , we canshow that it is possible to achieve ˜ O ( √ T ) regret and zero violation for large enough K (Theorem 3).
Conclusion and Future Work
We propose a model-free RL-based algorithm for linear MDP.
We have achieved ˜ O ( √ d 3 H 3 T )regret and ˜ O ( √ d 3 H 3 T ) constraint violation.
To the best of our knowledge, this is the first resultwhich shows ˜ O ( √ T ) regret and ˜ O ( √ T ) constraint violation without requiring a generator for themodel-free case.
We have extended the LSVI-UCB algorithm in the primal-dual type framework.
Wehave underlined the technical challenges in doing so and explained how the greedy policy fails toachieve an uniform concentration bound for individual value function.
Subsequently, we show that asoft-max type algorithm achieves that.
We also point out the fundamental analytical gap in existingworks that claim to achieve zero constraint violation without a simulator.
Whether we can achieve a better bound remains an open question.
Compared to [7], our regret is offby √ d factor which is due to the fact that we need to use uniform concentration bound.
The similargap is also observed for the unconstrained set-up as well.
Whether we can tighten this dependence ond remains an important future research direction.
Whether we can the tighten the dependence on Halso constitutes a future research direction.
Consideration of non-linear MDP also constitutes a futureresearch direction. Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] We have clearly compared ourresults with the existing work.
In the Conclusion and Future Work, we identified someimportant directions where our approach can be extended.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
We clearly
state the Assumptions 1 and 2 in Section
(b) Did you include complete proofs of all theoretical results?
[Yes] We have added theproofs in Appendix.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A](b)
Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[N/A]
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
) If your work uses existing assets, did you cite the creators?
[N/A]
Did you mention the license of the assets?
[N/A]
) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects
you include the full text of instructions given to participants and screenshots, if
applicable
you describe any potential participant risks, with links to Institutional Review
Board (
) approvals, if applicable
[N
Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]