A CHIEVING S UB - LINEAR R EGRET IN I NFINITE H ORIZON A VERAGE R EWARD C ONSTRAINED MDP WITH L INEAR F UNCTION A PPROXIMATION
Anonymous authors
Paper under double-blind review
A BSTRACT
We study the infinite horizon average reward constrained Markov Decision Process (CMDP).
In contrast to existing works on model-based, finite state space, we consider the model-free linear CMDP setup.
We first propose a computationally inefficient algorithm and show that ˜ O ( √ d 3 T ) regret and constraint violation can be achieved in which T is the number of interactions and d is the dimension of the feature mapping.
We also propose an efficient variant based on the primaldual adaptation of the LSVI-UCB algorithm and show that ˜ O (( dT ) 3 / 4 ) regret and constraint violation can be achieved.
This improves the known regret bound of ˜
O ( T 5 / 6 ) for the finite state-space model-free constrained RL which was obtained under a stronger assumption compared to ours.
We also develop an efficient policy-based algorithm via novel adaptation of the MDP-EXP2 algorithm to the primal-dual set up with ˜ O ( √ T ) regret and even zero constraint violation bound under a stronger set of assumptions.
I NTRODUCTION
In many standard applications of Reinforcement learning (RL) (e.g., autonomous vehicles), the agent needs to satisfy certain constraints (e.g., safety constraint, fairness).
These problems can be formulated as constrained Markov Decision process (CMDP) such that the agent needs to ensure that average utility (cost, resp.) exceeds a certain threshold (is below a threshold, resp.).
While CMDP in the finite state-space has been studied, those studies can not be extended to the large state space.
RL with value function approximation has demonstrated empirical success for large-scale
RL application using the deep neural networks.
However, theoretical understandings of constrained
RL with value function approximation is quite limited.
Recently, Ghosh et al.
(2022) has made some progress towards the understanding of the constrained RL for linear MDP under episodic setting.
In particular, Ghosh et al.
(2022) developed a primal-dual adaptation of the LSVI-UCB and showed ˜
O ( √ d 3 T ) regret and violation where d is the dimension of the feature space and T is the number of interactions.
Importantly, the above bounds are independent of the cardinality of state-space.
However, infinite-horizon model fits well compared to the finite horizon setting for many real-world applications (e.g., stock-market investment, routing decisions).
Compared to the discounted-reward model, maximizing the long-term average reward under the long-term average utility also has its advantage in the sense that the transient behavior of the learner does not really matter for the latter case (Wei et al., 2020).
Recently, model-based RL algorithm for infinite-horizon average reward CMDP has been proposed (Chen et al., 2022).
However, it considers tabular setup.
Further, the model-based approach requires large memory to store the model parameters.
It is also hard (computationally) to extend model-based approaches for infinite state-space such as linear MDP (Wei et al., 2020).
Model-free RL algorithm is more popular because of the ease of implementation, and being computational, and storage efficient particularly for the large state space.
However, model-free learning for the infinite horizon average reward setup is even more challenging.
For example, it is still unknown whether it is possible to achieve a computationally efficient model-free algorithm with ˜ O ( √ T ) regret even under the unconstrained tabular setup (Wei et al., 2020) for a weakly communicating MDP.
To the best of our knowledge, Wei et al.
(2022) is the only paper to study model-free algorithms for CMDPs in the infinite horizon average reward setup.
In particular, they consider the finite-state tabular setting and the regret scales polynomially with the number of states.
Thus, the result would not be useful for large-scale RL applications where the number of states could even be infinite.
To summarize, little is known for the performance guarantee of model-free algorithms in CMDPs beyond tabular settings in the infinite-horizon average reward setting, even in the case of linear CMDP.
Motivated by this, we are interested in the following question:
Can we achieve provably sample-efficient and model-free exploration for CMDPs beyond tabular settings for the infinite horizon average reward setting?
Contribution.
To answer the above question, we consider CMDP with linear function approximation, where the transition dynamics, the utility function, and the reward function can be represented as a linear function of some known feature mapping.
Our main contributions are as follows
• We propose an algorithm which achieves ˜ O ( √ d 3 T ) regret and constraint violation bounds with a high probability when the optimal policy belongs to a smooth function class (Definition 1).
The algorithm seeks to obtain weight vectors which parameterize the state-action bias functions by solving fixed-point equations.
Unlike the unconstrained setup, the greedy policy might not be optimal as it may not be feasible.
The algorithm thus searches for the policy which maximizes the average reward under the constraints.
We restrict the policy space to a smooth function class in order to obtain uniform concentration bound for value function class.
• Since the optimization problem is hard to solve, we also propose an efficient variant and show that ˜ O (( dT ) 3 / 4 ) regret and violation bound can be achieved.
Further, the optimal policy can be any stationary policy and need not to belong to a smooth policy function class.
The idea is to consider a finite-horizon episodic setup by dividing the entire time-horizon T in multiple episodes T/H where each episodes consists of H steps.
We then invoke the primal-dual adaptation of the LSVI-UCB algorithm proposed in Ghosh et al.
(2022) to learn the good policy for the finite-horizon setting.
Finally, we bound the gap between the infinite horizon average reward (constraint value, resp.) and the finite-horizon expected reward (constraint value, resp.)
to obtain the final result.
• We also propose an algorithm which can be implemented efficiently and achieves ˜ O ( √ T ) regret and ˜ O ( √ T ) constraint violation under a stronger set of assumptions (similar to the ones made in Wei et al.
(2021a)
for the unconstrained setup).
We, further, show that one can achieve zero constraint violation for large enough (still, finite) T while maintaining the same order on regret.
• We attain our bounds without estimating the unknown transition model or requiring a simulator , and they depend on the state-space only through the dimension of the feature mapping.
To the best of our knowledge, these sub-linear bounds are the first results for model-free (or, model-based) online RL algorithms for infinite-horizon average reward CMDPs with function approximations.
Wei et al.
(2022) proposes a model-free algorithm in the tabular setting which achieves ˜ O
( T 5 / 6 ) regret.
Since linear MDP contains tabular setting our result improves the existing results.
Further, we show that we can achieve zero constraint violation by trading the regret a little bit while maintaining the same order on regret under the same set of Assumptions of Wei et al.
1. O THER R ELATED W ORK
Model-free RL algorithms have been proposed (Xu et al., 2021;
Ding et al., 2020; Bai et al., 2021) to solve CMDP in the discounted infinite horizon setting.
All of these works consider an easier setting compared to standard RL in that they assume access to a simulator (Koenig and Simmons, 1993)
(a.k.a.
a generative model (Azar et al., 2012)), which is a strong oracle that allows the agent to query arbitrary state-action pairs and return the reward and the next state, hence greatly alleviating the intrinsic difficulty of exploration in RL.
Wei et al.
(2021b) proposed a ‘triple-Q’ algorithm which does not require a ‘simulator’.
However, it only considered tabular setting and the episodic set-up.
P RELIMINARIES
We consider an infinite horizon constrained MDP, denoted by ( S , A , P , r, g ) where S is the state space, A is the action space, P is transition probability measures, r , and g are reward and utility functions respectively.
We assume that S is a measurable space with possibly infinite number of elements, and A is a finite action set.
P ( ·| x, a ) is the transition probability kernel which denotes the probability to reach a state when action a is taken at state x .
We also denote P as p to simplify the notation.
p satisfies (cid:82) X p
( dx ′ | x, a ) = 1 (following integral notation from Hernández-Lerma (2012))
r : S ×
A → [0 , 1] , and g : S ×
A → [0 , 1] and are assumed to be deterministic.
We can readily extend to settings when r and g are random.
The process starts with the state x 1 .
Then at each step t ∈ [ T ] , the agent observes state x t ∈ S , picks an action a t ∈ A , receives a reward r ( x t , a t ) , and a utility g ( x t , a t ) .
The MDP evolves to x t +1 that is drawn from P ( ·| x t , a t ) .
In this paper, we consider the challenging scenario where the agent only observes the bandit information r ( x t , a t ) and g ( x t , a t ) at the visited state-action pair ( x t , a t ) .
The policy-space of an agent is ∆( A|S ) ; {{ π t ( ·|· ) } : π t ( ·| x ) ∈ ∆( A ) , ∀ x ∈ S , and t ∈ T } .
Here ∆( A ) is the probability simplex over the action space.
For any x t ∈ S , π t ( a t | x t ) denotes the probability that the action a t ∈ A is taken at step t when the state is x t .
We also denote the space of stationary policy as ∆ s ( A|S ) where the policy π ( ·| x t ) is independent of time t .
Let J πr ( x ) and J πg ( x ) denote the expected value of the average reward and average utility respectively starting from state x when the agent selects action using the stationary policy π ∈ ∆ s ( A|S )
[Equation]
for ⋄ = r, g , where E is taken with respect to the policy π and the transition probability kernel P .
From Puterman (2014), there exists a state-action bias function q satisfying the Bellman’s equation for ⋄ = r, g : q π ⋄ ( x, a ) + J π ⋄ ( x ) =
⋄ ( x, a ) + E x ′ ∼ p ( ·| x,a ) [ v π ⋄ ( x ′ )] where v π ⋄ ( x ) =
(cid:80) a π ( a | x ) q ⋄ ( x, a ) =
⟨ π ( ·| x ) , q ⋄ ( x, · ) ⟩ A .
q π
and v π are analogue to the Q -function and value function respectively for finite-horizon and discounted infinite horizon scenario.
The Problem : We are interested in solving the following problem
[Equation]
Let π ∗ be an optimal stationary solution of the above problem.
Note that both J πr and J πg depend on the initial state x .
Also note that our approach can be extended to multiple constraints, and the constraints of the form J πg ( x ) ≤ b .
Unlike the episodic set-up where the sub-linear regret is possible, it is known that even in the unconstrained set-up, the necessary condition for sub-linear regret is that the optimal policy has a long-term average reward and utility that are independent of the initial state (Bartlett and Tewari, 2012).
Hence, we assume the following:
Assumption 1.
There exists J ∗ r , J ∗ g and a stationary π ∗ such that they are solution of (2) and the followings hold for all ⋄ =
r, g , x ∈ S , and a ∈ A
[Equation]
We denote J ∗ r as the optimal gain.
We also denote the span of v ∗ ⋄ as sp( v ∗ ⋄ )
= sup s v ∗ ⋄ ( s ) − inf s v ∗ ⋄ ( s ) for ⋄ = r, g .
For a finite state and action space (a.k.a. tabular case), the weakly communicating
MDP is the broadest class to study the regret minimization in the literature, and is known to satisfy
Assumption 1.
Assumption 1 is adaptation of the Bellman’s optimality equation for unconstrained version (Wei et al., 2021a) in the CMDP.
Note that unlike the unconstrained case, here the optimal policy might not be greedy with respect to q r as the greedy policy might not be feasible.
Bellman’s equation with state-independent average reward and average utility : A stationary policy π satisfies the Bellman’s equation with state-independent average reward and average utility if there exists measurable functions v π ⋄ : S → R , q π ⋄ : S ×
A → R such that the following holds for all x ∈ S , ⋄ = r, g , and a ∈ A
[Equation]
Wei et al.
(2022) assumes that any stationary policy (not only the optimal policy) satisfies the
Bellman’s equation with state-independent average reward and average utility.
In Section 3 we consider the setup where Assumption 1 is satisfied which is weaker compared to Wei et al.
In
Section 4 we consider a set of assumptions which entails that every stationary policy satisfies (4).
Learning Metric : The agent is unaware of the underlying environment, and seeks to learn the optimal policy.
Hence, the agent can take action according to a non-stationary policy π t at a state x t .
We are interested in minimizing the following metrics
[Equation]
The regret denotes the difference between the total reward obtained by best stationary policy and the reward obtained by the learner.
The violation denotes how far the total utility achieved by the learner compared to the Tb .
Similar metrics are employed in model-based infinite horizon average reward (Chen et al., 2022) and in the finite horizon setup (Ding et al., 2021).
Linear Function Approximation.
To handle a possible large number of states, we consider the following linear MDPs.
Assumption 2.
The CMDP is a linear MDP with feature map ϕ : S ×
A → R d , if there exists d unknown signed measures µ = { µ 1 , . . . , µ d } over S such that for any ( x, a, x ′ ) ∈ S × A × S ,
[Equation]
and there exists vector θ r , θ g ∈ R d such that for any ( x, a ) ∈ S × A ,
[Equation]
Assumption 2 adapts the definition of linear MDP in Jin et al.
(2020) to the constrained case.
By the above definition, the transition model, the reward, and the utility functions are linear in terms of feature map ϕ .
We remark that despite being linear, P ( ·| x, a ) can still have infinite degrees of freedom since µ ( · ) is unknown.
Note that tabular MDP is part of linear MDP (Jin et al., 2020).
Similar Constrained Linear MDP model is also considered in Ghosh et al.
(2022) (for episodic setup).
Note that Ding et al.
; Zhou et al.
(2021) studied another related concept known as linear kernel MDP in episodic setup.
In the linear kernel MDP, the transition probability is given by P ( x ′ | x, a ) =
⟨ ψ ( x ′ , x, a ) , θ ⟩ .
In general, linear MDP and linear kernel MDPs are two different classes of MDP (Zhou et al., 2021).
Under linear CMDP, we can represent the q ∗⋄ function as a linear in the feature space.
Lemma 1 (Wei et al.
(2021a)) .
Under Assumptions 1 and 2, there exists a fixed w ∗⋄ ∈ R d such that q ∗⋄ ( x, a )
=
ϕ
( x, a ) T w ∗⋄ for ⋄ = r, g and all ( x, a ) ∈ S
× A with || w ∗⋄ ||√ d
(2 + sp( v ∗⋄ )) .
O PTIMISM - BASED A LGORITHMS 3. F IXED - POINT O PTIMIZATION WITH O PTIMISM
We present our first algorithm (Algorithm 1) which achieves ˜ O (sp( v ∗ r ) √ d 3 T ) regret and
O (sp( v ∗ g ) √ d 3 T ) constraint violation.
Note that for the unconstrained tabular version, Ω( √ T ) is the lower bound for regret (Auer et al., 2008).
Hence, our obtained bound is optimal in terms of regret (nearly).
Further, we show that we can attain zero violation while maintaining the same order in regret with an additional assumption (Remark 1).
In the following we describe our Algorithm and point out the key modifications we have made over the unconstrained version.
From Lemma 1 and (3), we have
[Equation]
Hence, the natural idea is to build an estimator w ⋄ ,t for w ∗⋄ from the data.
However, we do not know
J ∗⋄ , p , r , and g .
Further, we also need to find π in order to obtain v ∗⋄ (cf.(3)).
In the following, we describe how we address those challenges.
First, in order to handle unknown p , and ⋄ , we use the sample points till time t to estimate w ⋄ ,t .
In particular, if for a moment, if we assume J ∗ ⋄ and π are known then we can fit w ⋄ ,t such that ϕ ( x, a ) T w ⋄ ,t ≈ ⋄ ( x τ , a τ ) − J ∗⋄ + v ⋄ ,t ( x τ +1 ) .
Then solving a regularized least square problem with regularization term λ || w ⋄ ,t || 2 gives a natural estimate of w ⋄ ,t –
[Equation]
where Λ t = λ
I + (cid:80) τ<t ϕ ( x τ , a τ ) ϕ ( x τ , a τ ) is the empirical co-variance matrix.
Now, we replace v ⋄ ,t ( x τ +1 ) with the expression v ⋄ ,t ( x τ +1 )
= ⟨ π ( ·| x τ +1 ) , q ⋄ ,t ( x τ +1 , · ) ⟩ A which gives rise a fixed-point equation as q ⋄ ,t ( · , · ) =
ϕ ( · , · ) T w ⋄ ,t .
In order to incorporate uncertainty, we introduce a slack variable b ⋄ ,t with a bounded quadratic form || b ⋄ ,t || Λ t ≤ β ⋄ ( β ⋄ is a parameter for ⋄ = r, g ).
It will ensure that w ∗⋄ is contained within w ⋄ ,t + b ⋄ ,t with high-probability.
Now, we discuss how we handle the unknown J ∗⋄ and π .
We introduce J ⋄ ,t and π t as variables and design an optimization problem P 1 which maximizes J r,t with the constraint J g,t ≥ b .
We also restrict π to a class of soft-max policy Π (Definition 1) and search for π t over Π .
Definition 1.
We define the policy class as Π as the collection of soft-max policies Π
= { π ( ·|· ) ∝ exp( ϕ ( · , · ) T ζ ) , ζ ∈ R d , || ζ ||≤ L } .
We replace J ∗⋄ in (7) with J ⋄ ,t .
Further, we add the slack variable b ⋄ ,t to w ⋄ ,t in (7) to obtain (8) in
Algorithm 1.
This has similarity with the “optimism under uncertainty” as the optimal ( w ∗⋄ , b ∗⋄ , J ∗⋄ , π ∗ ) for ⋄ =
r, g are feasible for the optimization problem with high probability (Lemma 2) if π ∗ ∈ Π .
Hence, it naturally entails that J r,t ≥ J ∗ r .
Wei et al.
(2021a) also proposed an optimization problem with fixed-point equation (Algorithm 1) to obtain w ∗ r in the unconstrained setup.
In the following, we point out the main differences with the Algorithm 1 proposed in Wei et al.
(2021a)
and describe the necessity of the introduction of policy space Π .
First, we have an additional constraint where J g,t ≥ b since we consider CMDP.
Second and more importantly, in the unconstrained case (Wei et al., 2021a)
the policy was greedy with respect to q r .
Since the greedy policy may not be feasible, we need to search over policy in the CMDP. However, we restrict the policy to function class Π .
The main reason for introduction to Π is that the uniform concentration bound (a key step for proving both regret and violation for model-free approach) for value function class v ⋄ can not be achieved unless the policy has smooth properties such as Lipschitz continuous.
In particular, we need to show that log ϵ covering number for the class of v ⋄ scales at most O (log( T )) .
For the unconstrained case, we only need to obtain ϵ -covering number for v r and with greedy policy, ϵ -covering number for the class of q -function was enough.
However, we need to obtain ϵ -covering number for both v r and v g , hence, such a trick would not hold.
In order to obtain ϵ -covering number for v ⋄ , we use the smoothness of soft-max to show that log ϵ -covering number for the class of value function scales at most O (log( T ))
(Lemma 7).
In Appendix E.6 our analysis can be extended to the class of policies which satisfy Lemma 8.
Also note that we update the parameters when the determinant of Λ t doubles compared to Λ s t −where s t − 1 is the time step with the most recent update before the time t .
This can happen only at most O
( d log T ) times.
Similar to Wei et al.
(2021a), our algorithm also does not allow any efficient implementation due to the complicated nature of the optimization problem.
However, we can show that with high probability, the optimal solution and parameters are feasible of P 1 by the careful choice of the parameter β ⋄ , and using the fact that || w ∗⋄ ||≤ sp( v ∗⋄ ) √ d .
Lemma 2.
With probability 1 − 2 δ , Algorithm 1 ensures that J r,t ≥ J ∗ r if π ∗ ∈ Π .
Using Lemma 2 we obtain the regret and violation bound for Algorithm 1 in the following
Theorem 1.
With probability 1 − 5 δ , the regret and violation bounds are
[Equation]
if the optimal policy π ∗ ∈ Π .
This is the first result which shows that ˜ O ( √ T ) regret and violation is achievable for infinite horizon average reward linear CMDP under Assumption 1 using model-free approach .
Recently, Chen et al.
(2022) obtained ˜ O ( √ T ) regret and violation for model-based tabular setup for weakly communicating MDP.
The proposed algorithm (Algorithm 4) in Chen et al.
(2022) is also computationally inefficient.
Chen et al.
(2022) also requires the knowledge of sp( v ∗⋄ ) for ⋄ = r, g .
For the unconstrained MDP, the lower bound of regret is Ω( √ T ) (proved for tabular setup, (Auer et al., 2008)).
Hence, the regret order is optimal with respect to T .
We also show that it is possible to zero violation for large T under an additional set of Assumption (Remark 1).
1: Initialization: β ⋄ = 2 d (2 + sp( v ∗⋄ ))
(cid:112) log(32 Ld (2 + sp( v ∗⋄ ))
T/δ ) for ⋄ = r, g .
2: for t = 1 , . . .
, T do 3: if t = 1 or det(Λ t ) ≥ 2Λ s t − 1 then 4:
Set s t = t .
Obtain w ⋄ ,t for ⋄ = r, g as the solution of the following optimization problem P 1 :
[Equation]
Take action a t ∼ π t ( ·| x t ) and observe x t +1 .
Update
[Equation]
3. A LGORITHM BASED ON E PISODIC P RIMAL -D UAL A LGORITHM
We now present another optimism-based algorithm which can be implemented efficiently albeit with a sub-optimal regret and violation guarantee under the Slater’s condition (Assumption 3).
Assumption 3.
We assume the following, there exists ¯ π , v ¯ π , q ¯ π , J ¯ πr , and J ¯ πg such that (4) is satisfied; and J ¯ πg ≥ b + γ with γ > 0 .
The above assumption states that there exists a stationary policy with strict feasibility.
Strict feasibility assumption is common both in the infinite horizon average reward (Wei et al., 2022; Chen et al., 2022) and finite horizon episodic CMDP set up (Efroni et al., 2020; Ding et al., 2021).
We only need to know γ rather than the strictly feasible policy.
In algorithm 1 the inefficiency arises since we do not know how to efficiently solve a fixed-point equation and find the policy to solve the optimization problem.
We show that we eliminate the above inefficiencies by considering a finite-horizon problem.
In particular, we divide the entire horizon T into T/H episodes with H rounds and employ a finite-horizon primal dual optimistic LSVI-based algorithm proposed in Ghosh et al.
However, since the original problem is still infinite horizon we need to bound the gap because of the finite-horizon truncation which we describe in the following.
3.2. O UR A PPROACH
We replace the time index t with the combination of the episode index k and the step index h , t = ( k − 1) H
+ h .
The k -th episode starts at time ( k − 1) H + 1 .
The agent chooses the policy π k h for the step
h ∈
[ H ] at episode k ∈ [ K ] .
Note that for finite-horizon setting, the policy needs not to be stationary even though the optimal policy for the infinite horizon problem is stationary.
As we mentioned, the idea is here to obtain policy with good performance guarantee in the episodic setup and then bound gap between the infinite horizon and episodic setup.
We now introduce the notations specific to the episodic setting.
[Equation]
  .
Note that since J πg ≥ b , the natural choice of the constraint in the episodic CMDP would be
V π g, 1 ≥ Hb .
However, in order to compare the best policy in the episodic setup with the optimal policy in the infinite horizon setup we need to ensure that the optimal stationary policy for the infinite horizon problem is feasible in the episodic setup.
If we have V πg, 1 ≥ Hb − sp( v ∗ g ) we can conclude that the optimal stationary policy π ∗ is also feasible for the episodic setup using the following lemma:
Lemma 3.
If J π ⋄ ( s ) =
J π ⋄ for ⋄ = r, g , and any state s , stationary policy π , then
[Equation]
Further, in order to apply the primal-dual algorithm we also need to show that there exists a strictly feasible policy for the episodic setup.
Again using Lemma 3 and from Assumption 3, we conclude that there exists a strictly feasible policy for the episodic setup if V πg, 1 ≥ Hb + γ − sp( v ¯ πg ) .
Thus, we consider the following constraint V πg, 1 ≥ Hb − κ where κ = max { sp( v ∗ g ) , sp( v ¯ πg ) − γ } .
Lagrangian : We consider the composite value function as V π,Yh (
[Equation]
The following is proved in Ding et al.
Lemma 4 (Boundedness of Y ∗ ) .
The optimal dual-variable Y ∗ ≤ V ˜ π
[Equation]
Hγ
[Equation]
H
Hγ where ¯ π ∗ is the optimal solution of the episodic CMDP.
Definition 2.
We set ξ = 2 /γ .
ξ is used to truncate the dual variable.
3.2. A LGORITHM
We now describe our proposed Algorithm 2 (Appendix C).
First, note that the finite-horizon stateaction value function for linear MDP can be represented as the following linear form
[Equation]
[Equation]
Hence, the natural estimates for the parameters w kr,h , w kg,h are obtained by solving the following regularized least square problem
[Equation]
Q k ⋄ ,h ( x, a ) =
⟨ w k ⋄ ,h , ϕ
( x, a ) ⟩ + β || ϕ
( x, a ) || Λ − 1 k where β || ϕ
( x, a ) || Λ − 1 k is the bonus term.
Since we are considering a finite-horizon, hence, we can recursively set V ⋄ ,h +1 starting from V ⋄ ,
H +
= 0 .
Thus, we do not need to solve the fixed-point equation unlike in Algorithm 1.
Another difference with
Algorithm 1 is that we introduce a pointwise bonus term for optimism.
Finally, we use a primal-dual adaptation unlike in Algorithm 1.
The policy is based on the soft-max.
In particular, at step h , π h,k ( a | x ) is computed based on the soft-max policy on the composite Q -function vector { Q kr,h ( x, a ) + Y k Q kg,h ( x, a ) } a ∈A where Y k is the lagrangian multiplier.
The computation of policy is also efficient unlike in Algorithm 1 as we do not need to search over the policy space.
Note that the greedy policy with respect to the composite
Q -function vector fails to show that log ϵ -covering number for the individual value function class is at most log( T ) (Ghosh et al., 2022).
Since the policy is no longer greedy, we can not conclude the optimism with respect to the combined Q -function.
Rather, the parameter α is chosen in order to balance between optimism and the uniform concentration bound.
The last part (Step 15) of Algorithm 2 includes dual-update step.
The dual variable increases if the estimated utility value function does not satisfy the constraint.
This algorithm is based on Ghosh et al.
However, there is an important difference.
Ghosh et al.
(2022) maintains different covariance matrix at each step h , Λ kh .
But we maintain a single Λ k for all h since the transition probability and reward are the same across the steps.
3.2. M AIN R ESULTS
Theorem 2.
Under Assumptions 1, 2, and 3, with probability 1 − 5 δ , the regret and constraint violations are
[Equation]
where ι = log(log( |A| ) ξ 2 dT/δ ) .
Unlike in Theorem 1, here, the optimal policy does not need to belong to Π .
Note that Wei et al.
(2022) proved that the regret bound is
˜ O
( T 5 / 6 ) for the model-free constrained RL algorithm in the tabular regime under a much stronger Assumption compared to ours.
Since linear MDP contains tabular setup, our approach improves the known upper bound for the tabular case as well.
We show that it is possible to achieve zero violation while maintaining the same order on regret under an additional Assumption (Remark 1).
Note that the regret is also ˜ O (( dT ) 3 / 4 ) for the unconstrained linear MDP (Wei et al., 2021a) for the efficient implementation variant of the algorithm.
Analysis : The overall regret can be decomposed in
Regret( T )=
[Equation]
The third term in the right hand side can be bounded using the Azuma-Hoeffding inequality.
Hence, we focus on the first two terms of the right-hand side.
The second-part is exactly same as the regret for the episodic scenario.
Thus, from the analysis of Ghosh et al.
(2022), we show that the above is upper bounded by ˜ O ( √ d 3 TH 2 ) .
Note that compared to Ghosh et al.
, we have √ H improvement as in our case the transition probability is independent of the time h unlike in Ghosh et al.
One may wonder why not set the length H too small.
However, since our original problem is about average-reward over infinite horizon, we need to argue that the best finite-horizon policy also performs well under the infinite-horizon criteria.
The first term characterizes the gap.
We show that the sub-optimality gap to the best finite horizon policy is bounded by T sp( v ∗ r ) /H using Lemma 3.
Hence, if we set too small H , the gap would increase.
By setting H = T 1 / 4 /d 3 / 4 we balance the upper bounds on the first two terms of the right-hand side.
As T increases, H also increases.
Constraint Violation : The violation term can also be decomposed as follows
[Equation]
The first term is exactly the constraint violation in the finite-horizon episodic case.
Using the analysis of Ghosh et al.
(2022), we obtain the upper bound of the first term as ˜ O ( √ d 3 H 2 T ) .
The second term can be bounded by Azuma-Hoeffding inequality.
The third term denotes the error incurred due to the introduction of κ .
We obtain the final constraint violation bound by replacing H with T 1 / 4 /d 3 / 4 .
P OLICY -B ASED A LGORITHM
Even though Algorithm 2 is computationally efficient, still one needs to compute V k ⋄ ,h +1 for every encountered state and needs to evaluate (Λ k ) − 1 .
We now show that it is possible to achieve ˜ O ( √ T ) regret and constraint violation under a different and relaxed set of assumptions using a computationally less intensive algorithm.
We propose a policy-based algorithm towards this end.
First, we state the assumptions and then we describe the algorithm and the results.
Assumption 4.
There exists a constant t mix ≥ 1 such that for any policy π , and any distribution ν 1 , ν 2 ∈ ∆ X over the state-space ||
P π ν 1 − P π ν 2 || TV ≤ e − 1 /t mix || ν
1 − ν 2 || TV where P π ν ( x ′ ) =
(cid:82)
X (cid:80) a π ( a | x ) P ( x ′ | x, a ) dν ( x ) , TV is the total variation distance.
Ergodic MDP satisfies the above assumption Hao et al.
With the above assumption, we can conclude the following, as shown in the unconstrained setup (Wei et al., 2021a)).
Lemma 5.
Under Assumption 4, any stationary policy π satisfies (4).
Hence, under Assumption 4, the gain is constant, and any stationary policy indeed satisfies (4).
Assumption 5.
Let λ min ( · ) be minimum eigen value of a matrix.
There exists σ > 0 such that for any π , λ min
[Equation]
The assumption intuitively guarantees that every policy is explorative in the feature space.
similar assumptions are also made (Wei et al., 2021a; Abbasi-Yadkori et al., 2019) in the unconstrained case.
Lagrangian : J πr + Y J πg is the Lagrangian for the stationary policy π where Y is the dual variable.
Under Assumption 3, optimal dual-variable Y ∗ ≤ J π
[Equation]
4. A LGORITHM
We consider a primal-dual adaptation of MDP-EXP2 in Algorithm 3 (Appendix D).
The algorithm proceeds in epochs where each epoch consists of B =
˜
O ( dt mix /σ ) steps.
Within each epoch, we collect B/ 2 N trajectories
where N = ˜ O
( t mix ) .
The update of policy π k for k -th epoch is equivalent to the online mirror descent (OMD), i.e.,
[Equation]
where q k − 1 ⋄ ( x, a ) =
( w k − 1 ⋄ ) T ϕ ( x, a ) .
We consider a composite q -function which is the sum of the q r and q g (scaled by the dual-variable).
We estimate w k ⋄ from sample average reward R k,m and sample average utility G k,m by utilizing Lemma 5 and the fact that q π ⋄ is linear in feature space.
Since we consider primal-dual adaptation of MDP-EXP2 in Wei et al.
(2021a)
, we maintain two different types of sample average for the utility.
G k,m is used to estimate q π k g + NJ π k g ; on the other hand, ˆ J k is used to estimate J π k g .
The dual update is based on the value of ˆ J k .
4. M AIN R ESULTS
Theorem 3.
Under Assumptions 2, 4,5, and 3, Algorithm 2 ensures that with probability 1 − 4 δ ,
Regret [Equation]
[Equation]
Wei et al.
(2021a) shows that ˜ O ( √ T ) regret is achieved in the expectation under Assumptions 4, and 5 in the unconstrained setup.
We show that even with a high probability, such a regret can be achieved along with ˜ O ( √ T ) of violation.
Assumption 2 can be relaxed to the scenarios where q π ⋄ is linear (even though the underlying MDP is not linear) similar to unconstrained setup (Assumption 4 in Wei et al.
(2021a)).
Hence, ˜ O ( √ T ) regret and violation can be achieved for any linear q π even if the underlying MDP is not linear.
In the model-based tabular CMDP, Chen et al.
(2022) shows that the regret also scales with ˜ O ( √ T ) in the ergodic MDP.
Also note that the regret scales with t mix and ξ there as well (order of t mix is worst there).
Note that the dependence on d is implicit as 1 /σ
= Ω( d ) .
We need to know t mix , and σ in Algorithm 3.
Similar to the unconstrained setup Wei et al.
(2021a), we can modify our algorithm to incorporate unknown t mix and σ with slightly worsened regret.
Remark 1.
We can reduce the violation to 0 with sacrificing the regret a little bit (the order is the same) under the Assumptions in Section 4 (Appendix G.5).
The idea is the following, we consider a tighter optimization problem where we add ϵ in the right-hand side of the constraint in (2).
We then bound the difference of the optimal gains in the tighter problem and the original problem as a function of ϵ .
The bound on regret and violation can be attained following our analysis for the tighter optimization problem since it is a CMDP with b
+ ϵ in place of b .
Hence, by carefully choosing ϵ , we can show that it is strictly feasible, and achieves zero violation for large enough T (Lemma 36).
For the assumption in Section 3.2 and Section 3.1, we need an additional assumption (any stationary policy satisfies Bellman’s equation (cf.(4)) to obtain zero constraint violation (Appendix G.5).
The idea is similar to the one as described in earlier paragraph.
The additional assumption would help us to bound the difference between the tighter and the original problem.
E XPERIMENTAL R ESULTS
We conduct numerical experiments to validate our theory on an environment similar to Chen et al.
We implement Algorithm 3 and observe that regret and violation indeed grow sub-linearly.
Please see Appendix H for details.
C ONCLUSION AND F UTURE W ORK
In this work, we provide model-free RL algorithms under two different sets of Assumptions.
To the best of knowledge, ours is the first work on the average reward constrained linear MDP.
Our results improve the regret and violation bound even for the finite horizon tabular model-free constrained RL setup.
Whether we can develop a computationally efficient algorithm which achieves optimal regret under Assumption 1 constitutes a future research question.
Whether we can extend the analysis to the non-linear function approximation is also an open question.
A M ORE R ELATED W ORK
To develop online sample-efficient algorithms for CMDPs, prior works have largely resorted to the finite horizon setup (Efroni et al., 2020;
Brantley et al., 2020; Kalagarla et al., 2020; Liu et al., 2021)
[Table]
Model-based RL in the episodic linear kernel CMDP is considered in Ding et al.
Model-based infinite horizon average reward CMDP for tabular setup is also considered in Zheng and Ratliff (2020); Singh et al.
B S UMMARY OF O UR R ESULTS IN A T ABLE
[Table] [Table] [Table] [Table]
[Table]
[Table]
C A LGORITHM
: Initialization:
[Equation]
2: for episodes k = 1 , . . .
, K do 3:
Receive the initial state x k 1 .
4: Λ k ← (cid:80) k − 1 τ =1 (cid:80)
h ϕ ( x τh , a τh ) ϕ
( x τh , a τh ) T
+ λ I 5: for step
h = H, H − 1 , . .
, 1 do 6: w k r,h ← (Λ k ) − 1 [ (cid:80) k − 1 τ =1 (cid:80) H h
′ =
1 ϕ ( x τ h ′ , a τ h )
[ r ( x τ h ′ , a τ h ′ )
+ V k r,h +1 ( x τ h ′ +1 )]] 7: w kg,h ← (Λ k ) − 1 [ (cid:80)
k − 1 τ =1 (cid:80)
Hh ′ =
1 ϕ ( x τh ′ , a τh ′ )
[ g ( x τh ′ , a τh ′ )
+ V kg,h ′ +
( x τh ′ +1 )]] 8:
Q kr,h ( · , · ) ← min {⟨ w kr,h , ϕ ( · , · ) ⟩ + β ( ϕ ( · , · ) T (Λ k ) − 1 ϕ ( · , · )) 1 / 2 , H } 9:
Q kg,h ( · , · ) ← min {⟨ w kg,h , ϕ ( · , · ) ⟩ + β ( ϕ ( · , · ) T (Λ k ) − 1 ϕ ( · , · )) 1 / 2 , H } 10: π h,k ( a |· ) = exp( α ( Q kr,h ( · , a ) + Y k Q kg,h ( · , a )))
(cid:80) a exp( α ( Q kr,h ( · , a ) + Y k Q kg,h ( · , a ))) 11:
V kr,h ( · ) =
(cid:80) a π h,k ( a |· ) Q kr,h ( · , a ) , V kg,h ( · ) =
(cid:80) a π h,k ( a |· ) Q kg,h ( · , a ) 12: for step h = 1 , . . .
, H do 13:
Compute Q kr,h ( x kh , a ) , Q kg,h ( x kh , a ) , π ( a | x kh ) for all a .
Take action a kh ∼ π h,k ( ·| x kh ) and observe x kh +1 .
Y k +1 =
max { min { Y k + η ( b − V k g, 1 ( x k 1 )) , ξ } , 0 }
1: Initialization: Y 1 = 0 , N = 8 t mix log( T ) w ⋄ , 1 = 0 , B = 32 N (log( dT )) /σ α = min { 1 / (1 + ξ ) √ Tt mix , σ/ (24 N (1 + ξ )) } , η = ξ/ (cid:112) T/B .
2: for epochs k = 1 , . . .
, T/B do 3:
Define the policy π k ( a | x ) such that π k ( a | x ) ∝ exp (cid:16) αϕ ( x, a ) T ( (cid:80) k − 1 j =1 w r,j + Y j w g,j ) (cid:17) , and initialize count ¯ J k .
Execute π k in the entire epoch.
5: for t = ( k − 1) B + 1 , . .
, kB do 6:
Execute a t ∼ π k ( ·| x t ) , observes, r t ( x t , a t ) , g t ( x t , a t ) , and x t +1 .
7: if t ≥ ( k − 1) B + N + 1 then 8:
Sum the constraint ¯ J k
=
¯ J k + g ( x t , a t ) .
9: for step m = 1 , . . .
, B/ 2 N do 10:
Define τ k,m =
( k − 1) B + 2 N ( m − 1)
+ N +as the start of the m -th trajectory.
Compute
R k,m = τ k,m + N − 1 (cid:88) t = τ k,m r ( x t , a t )
G k,m = τ k,m + N − 1 (cid:88) t = τ k,m g ( x t , a t ) 12:
Compute
M k =
B/ 2 N (cid:88) m = (cid:88) a π ( a | x τ k,m ) ϕ
( x τ k,m , a τ k,m ) ϕ
( x τ k,m , a τ k,m ) T 13: if λ min ( M k ) ≥ Bσ/ (24 N ) , then 14:
[Equation] [Equation]
N
[Equation]
x
[Equation]
a
[Equation]
k,
[Equation]
G k,m 16: else 17:
[Equation]
g
[Equation]
k = 0 .
[Equation] [Equation]
E P ROOF OF THE R ESULT OF S ECTION 3.
Notations : Without loss of generality, we assume that the first element of ϕ ( · , · ) is 1 .
Hence ϕ ( · , · ) T e 1 = 1 where e 1 is a d -dimensional vector where the first element is 1 and the rest are 0 .
Also recall that || w ∗⋄ ||≤ √ d (2 + sp( v ∗⋄ )) from Lemma 1.
As shown in Wei et al.
(2021a)
we have sup x | v ∗⋄ ( x ) |≤ 1 2sp( v ∗⋄ ) without loss of generality.
Without loss of generality, we assume ||
ϕ
( x, a ) || 2 ≤ √ 2 for all ( x, a ) ∈ S × A , || µ ( S )
|| 2 ≤ √ d , ||
θ j || 2 ≤ √ d for ⋄ = r, g .
E. P ROOF -S KETCH OF T HEOREM
Note from (3) that for ⋄ = r, g
[Equation] [Equation]
We then show that
Lemma 6.
With probability 1 − δ ,
[Equation]
where s t is the last time the optimization problem is solved before time t .
( J s t , ⋄ , b s t , ⋄ , w s t , ⋄ , π s t ) are the solutions which also specify q s t , ⋄ and v s t , ⋄ .
The key step in proving the above lemma is that we need to show
[Equation]
is upper bounded by O (log T ) with high probability.
To this end, value-aware uniform concentration is required to handle the dependence between v s t , ⋄ and samples { x τ } k − 1 τ =1 , which renders the standard self-normalized inequality infeasible in the model-free setting.
The general idea here is to fix a function class V ⋄ in advance and then show that each possible value function in our algorithm v s t , ⋄ is within this class which has polynomial log ϵ -covering number.
The idea is to show that the function class q s t , ⋄ has log ϵ covering number which is at most O ( d log(1+ sp( v ∗⋄ ) /ϵ ))
(Lemma 7).
Now, using the smoothness property of soft-max, we can also show that the policy class also has log ϵ -covering number which is at most d log(1 + 16 L/ϵ ) .
Combining the above we compute the ϵ -covering number for the class V ⋄ and show that (16) is upper bounded by log( T ) with high probability.
The detailed proof is in Lemma 7.
Now, we divide the proof of Theorem 1 in two parts.
First, we bound the regret and subsequently, we bound the violation.
Proof of Regret : From Lemma 2 with probability 1 − 2 δ , w ∗ r , w ∗ g , J ∗ g , J ∗ r , π ∗ are feasible solution for the optimization problem.
Since we are maximizing J r , thus, J s t ,r ≥ J ∗ r with probability 1 − 2 δ .
Further, from Lemma 6 with probability 1 − δ ,
[Equation]
Hence, from union bound, with probability 1 − 3 δ , (cid:88) t
[Equation]
By rearranging
T
[Equation]
≤ [Equation] We define F t, 2 as the σ -algebra generated by [( x τ , a τ )]
tτ =1 ∪ x t +1 .
Then, E [ v s t , ⋄ ( x t ) − q s t , ⋄ ( x t , a t ) |F t − 1 ] = 0 .
Thus, v s t , ⋄ ( x t ) − q s t , ⋄ ( x t , a t ) is a Martingale difference.
Since || w s t ,r ||≤ √ d (2 + sp( v ∗ r )) (from the optimization problem in Algorithm 1), hence, v s t ,r ( x t ) − q s t ,r ( x t , a t ) is upper bounded by β r (cid:112) T log( T/δ ) with probability 1 − δ .
Thus, from union bound, with probability 1 − 4 δ , we have
[Equation]
Notice that every time the algorithm updates (i.e., s t ̸
= s t − 1 ) it holds that det (Λ t ) =
det(Λ s t ) ≥ 2det(Λ s t − 1 ) .
Since det (Λ T +1 ) / det(Λ 1 ) ≤ (1 + T/λ ) d , thus, it can not happen more than d log 2 (1 +
T )
=
O ( d log T ) times.
Thus, with probability 1 − 4 δ , T
[Equation]
The first term in the right-hand side of (19) can be bounded by O (sp( v ∗ r ) (cid:112) dT log( T/δ )) with probability 1 − δ by Azuma-Hoeffding inequality.
Hence, with probability 1 − 5 δ ,
[Equation]
where the last step holds via Azuma-Hoeffding inequality.
Hence, the result follows from the definition of β r , and (14).
Proof of Constraint Violation : From Lemma 6 with probability 1 − δ ,
[Equation]
Similar to (19) we obtain with probability 1 − 3 δ
[Equation]
Hence, rearranging (20) and from (14), we obtain T
[Equation]
[Equation]
Note that we can not conclude that J s t ,g ≥ J ∗ g unlike the reward term.
Nevertheless, since J s t ,g is feasible, thus, from the optimization problem P 1 in Algorithm 1, we have J s t ,g ≥ b .
Hence, from (21) with probability 1 − 3 δ ,
[Equation]
Hence, the result follows.
E. P ROOF OF L EMMA
For ⋄ = r, g ,
[Equation] [Equation]
where ϵ
[Equation]
Let us denote ε ∗ τ, ⋄ =
E x ′ ∼ p ( ·| x τ ,a τ ) v ∗⋄ ( x ′ ) − v ∗⋄ ( x τ +1 ) which is Martingale difference, then from the self-concentration result with probability 1 − 2 δ
(Theorem 4),
[Equation]
Now, det (Λ t ) ≤ ( λ + t ) d , det (Λ 0 ) ≥ λ d .
Thus,
[Equation]
On the other hand, note that λ (Λ t ) − 1 w ∗⋄ ,t ≤ β ⋄ / 2 , hence, by selecting b ⋄ ,t = λ (Λ t ) − 1 w ∗⋄ ,t + ϵ ∗⋄ ,t (which is upper bounded by β ⋄ ) and
J ∗ g ≥ b we have the result.
E. P ROOF OF L EMMA
Proof.
For notational simplicity, we denote s t =
s .
From the definition of w s, ⋄
[Equation]
where ϵ s, ⋄ is defined in (23)
[Equation]
Therefore,
[Equation]
where
[Equation]
We now bound the third term in the right hand side which will prove the lemma.
From Lemmawith probability 1 − δ
[Equation]
We now compute N ϵ, ⋄ which is the ϵ -covering number for the value function class v s, ⋄ .
Lemma 7.
The ϵ -covering number for the class V ⋄ = { v ⋄ | v ⋄ ( x ) = ⟨ π ( ·| x ) , q ⋄ ( x, · ) ⟩ A is upper bounded by
[Equation] [Equation]
where ϵ ′ =
ϵ 1 + (2 + sp( v ∗⋄ )) √ 2 d , π ∈ Π , and q ⋄ ( · , · ) =
w T ⋄ ϕ ( · , · ) , where
[Equation]
Proof.
Proof is in Appendix E.4.
Hence, from (29) and putting ϵ = 1 T , we obtain
[Equation]
31) with probability at least 1 − δ .
Now,
[Equation]
Further, || b s, ⋄ || Λ s ≤ β ⋄ .
Hence,
[Equation]
Thus,
T (cid:88) t = ( q
[Equation]
( x t
[Equation]
a t )
[Equation]
a
[Equation]
≤
[Equation]
(cid:88)
[Equation]
[Equation]
E x
[Equation]
p (
[Equation]
x t
[Equation]
a t )
[Equation]
v s t
[Equation]
⋄ ( x ′
[Equation]
− v ∗⋄ ( x ′ )]
+ O
[Equation]
β ⋄
T
[Equation]
t
[Equation]
|| ϕ
( x t
[Equation]
a t ) || Λ
[Equation]
t ) ≤
T (cid:88) t =
( J ∗⋄ − J s t , ⋄ )
+
T (cid:88) t =
E x ′ ∼ p ( ·| x t ,a t )
[ v s t , ⋄ ( x ′ ) − v ∗⋄ ( x ′ )]
+ O  
 β ⋄ √ T (cid:118)(cid:117)(cid:117)(cid:116)
[Equation]
where the second inequality follows from Cauchy-Schwarz inequality.
The last inequality follows from the fact that (cid:113)(cid:80) Tt =
1 || ϕ
( x t , a t )
|| 2Λ − 1 t ≤ (cid:112) 2 log(det(Λ T +1 ) / (det(Λ 1 ))) , and log(det(Λ T +1 ))
≤
d log( λ + T )
E. P ROOF OF L EMMA
Proof.
|| w s, ⋄ ||≤ (2 + sp( v ∗⋄ ))
√ d .
By Lemma 22, the covering number for w s, ⋄ is
[Equation]
Since q s, ⋄ ( · , · ) = ϕ ( · , · ) T w s, ⋄ and || ϕ ||≤ √ 2 , ϵ -covering number for w s, ⋄ is also the ϵ -covering number for the class of state-action bias functions q s, ⋄ is
[Equation]
with respect to ∞ -norm.
For the policy-parameter class, note that || ζ ||≤ L .
Hence, the ϵ -covering number for the parameter class is (1 + 2 L/ϵ ) d from Lemma 22.
We also use the following result where we show that the 1 -norm of the difference between two policies is close if the parameters are close from the property of soft-max.
Lemma 8.
If π, ˜ π ∈ Π where π is parameterized by ζ and ˜ π is parameterized by ˜ ζ , then for any x ∈ X (cid:88)
[Equation]
(35) if || ζ − ˜ ζ ||≤ 1 / 2 .
Proof.
See Appendix E.5.
The value function class v s, ⋄ is parameterized by ( ζ, w s, ⋄ ) .
Consider ˜ ζ such that ||
˜ ζ − ζ ||≤ ϵ/ 8 , then from Lemma 8 for every state x (cid:88)
[Equation]
Further, consider ˜ w s, ⋄ such that || w s, ⋄ − ˜ w s, ⋄ ||≤ ϵ , then max x,a | q s, ⋄ ( x, a ) − ˜ q s, ⋄ ( x, a ) |≤ ϵ where ˜ q s, ⋄ is parameterized by ˜ w s, ⋄ .
Now, ˜ v s, ⋄ = ⟨ ˜ π, ˜ q s, ⋄ ⟩ , and v s, ⋄ = ⟨ π, q s, ⋄ ⟩ .
Then, for any x ,
[Equation]
Thus, consider ϵ ′ =
ϵ
[Equation]
, then,
[Equation] [Equation]
E. P ROOF OF L EMMA
Proof.
Dividing π ( a | x ) by ˜ π ( a | x ) yields
[Equation]
Since || ϕ ||≤
[Equation]
and exponential is a strictly increasing function
[Equation]
thus,
[Equation]
where the first inequality stems from the fact that (cid:80) a ′ π ( a ′ | x ) = 1 .
The second inequality stems from the fact that e x ≤ 1 + 2 x if x ∈ (0 , 0 . 5) .
Hence,
[Equation] [Equation]
We can apply the similar argument with the roles ζ and ˜ ζ reversed to conclude
[Equation] [Equation]
Hence,
[Equation]
E. C LASS OF P OLICIES
Note that we only use Lemma 8 to show Lemma 7 for the class of bias value function.
Thus, our analysis can be extended to any policy parameterized by ζ such that
Definition 3.
For any x , (cid:80) a | π ( a | x ) − ˜ π ( a | x ) |≤ R || ζ − ˜ ζ || holds for some || ζ − ˜ ζ ||≤ L 1 where R is constant and L 1 > 0 .
Obviously, soft-max policy is one such policy which satisfies the condition of the Definition 3.
F P ROOF OF THE R ESULTS OF S ECTION 3.
Organization : In Appendix F.1, we provide the proof sketch and subsequently we prove various parts in Appendix E.2 to Appendix E.9.
We formally prove Theorem 2 in Appendix F.10.
Notations for this Section : Throughout this section, we denote Q kr,h , Q kg,h , w kr,h , w kg,h , Λ kh as the Qvalue and the parameter values estimated at the episode k .
V kj,h ( · ) = ⟨ π h,k ( ·|· ) , Q kj,h ( · , · ) ⟩ A .
π h,k ( ·| x ) is the soft-max policy based on the composite Q -function at the k -th episode as Q kr,h + Y k Q kg,h .
To simplify the presentation, we denote ϕ
kh
=
ϕ
( x kh , a kh ) .
Without loss of generality, we assume ||
ϕ
( x, a ) || 2 ≤ 1 for all ( x, a ) ∈ S × A , || µ ( S ) || 2 ≤ √ d , ||
θ j || 2 ≤ √ d for ⋄ = r, g .
F. P ROOF S KETCH OF T HEOREM
We reiterate the decomposition of regret.
[Equation]
The first term is bounded via Lemma 3 by ( T/H )sp( v ∗ r ) which we prove in Appendix F.3.
The third-term is bounded using Azuma-Hoeffding inequality
[Equation]
and its upper bound is given by O ( √ THι ) with probability 1 − δ .
We bound the second term in Appendices F.4–F
[Equation]
In the following, we describe a rough idea to bound the second term.
Similar to Efroni et al.
[Equation]
we first establish the following decomposition, which upper bounds the sum of regret and violation (Lemma 9).
Lemma 9.
For any Y ∈ [0 , ξ ]
[Equation]
we have
[Equation]
Proof.
See Appendix F.4.
This will serve as the basis when applying optimization tools to bound the violation as well.
Note that by making Y = 0 , we can bound the regret.
Note that T 1 is similar to the term related to optimism in the unconstrained case with the difference being that we now have two value functions weighted by the dual variable Y k .
Similarly, T 2 is similar to prediction error term with the additional weight by Y .
Since the first term in the above inequality can be easily bounded with a proper choice of η , we are only left to bound T 1 and T 2 , respectively.
In order to bound T 1 and T 2 , we use the following result
Lemma 10.
There exists a constant C 2 such that for any fixed p ∈ (0 , 1) , if we let E be the event that
[Equation]
for all j ∈ { r, g } , χ = log[2( C 1 + 1) ξ log( |A| ) dT/δ ] , for some constant C 2 , then Pr( E ) = 1 − 2 δ .
Proof.
Please see the proof of Lemma 8 in Ghosh et al.
The above lemma entails that for all ( k, h ) ∈
[ K ] ×
[ H ] with high probability
[Equation]
is upper bounded by lower order term (e.g., O ( d √ log T ) .
Similar to Algorithm 1 the general idea here is to fix a function class V ⋄ ,h in advance and then show that each possible value function in our algorithm V k ⋄ ,h is within this class which has log-covering number which scales with log T .
Similar to Ghosh et al.
we introduce soft-max policy and define the following corresponding function classes to show that the value function class has log ϵ -covering number which scales with log T .
We first define the following class for Q -function for ⋄ = r, g .
Q ⋄ = { Q ⋄ | Q ⋄ ( · , · ) = min {⟨ w ⋄ , ϕ ( · , · ) ⟩ + β (cid:112) ϕ ( · , · ) T (Λ h ) − 1 ϕ ( · , · ) , H } .
Then, we define the following value function class V ⋄ .
V ⋄ =
{ V ⋄ | V ⋄ ( · ) =
(cid:80) a π ( a |· ) Q ⋄ ( · , a ); Q ⋄ ∈ Q ⋄ , π ∈ Π e } , where Π e is given by the following class Π e = { π | π ( a |· ) =
S OFT -M AX aα (( Q r ( · , · ) + Y Q g ( · , · )); ∀ a ∈ A , Q r ∈ Q r , Q g ∈
Q g , Y ∈ [0 , ξ ] } .
As described in Ghosh et al.
(2022) greedy algorithm with respect to the composite
Q -function fails to show that the log ϵ -covering number for individual V ⋄ scales at most O (log( T )) as the greedy policy is not Lipschitz.
Using the above we show the following
Lemma 11.
There exists an absolute constant β = C 1 dH √ ι , ι = log(log( |A| ) ξ 2 dT/δ ) , and for any fixed policy π , on the event E defined in Lemma 10, we have
[Equation]
(47) for some ∆ k ( x, a ) that satisfies | ∆ k
( x, a ) |≤ β (cid:112) ϕ
( x, a ) T (Λ k ) − 1 ϕ
( x, a ) .
Proof.
See Appendix F.5.
The above lemma bounds the difference between the value function maintained in Algorithm(without the bonus term) and the value function for any policy for both the reward and utility value functions.
We bound this using the expected difference at the next step plus an error term.
The result shows that this error term can be upper bounded by the bonus term with a high-probability.
Using the above, we can easily show the following–
Lemma 12.
With prob.
1 − 2 δ , (for the event in E )
[Equation]
Proof.
See Appendix F.6.
Using the above, and the property of the soft-max (Lemma 19), we obtain
Lemma 13.
With probability 1 − 2 δ , T 1 ≤ H log( |A| )
α
Proof.
See Appendix F.7.
Further, from Lemma 12, we obtain the following
Lemma 14.
On the event defined in E in Lemma 10, we have
[Equation]
where
[Equation]
Proof.
See Appendix F.
[Equation] Using the above result, Azuma-Hoeffding inequality, and plugging the value of β we obtain the following
Lemma 15.
With probability 1 − 4 δ , T 2 ≤ ( Y + 1) O ( √ d 3 TH 2 ι 2 )
Proof.
See Appendix F.9.
Combining all the pieces: In (45), we replace Y with 0 , and η = ξ/ √ KH 2 .
Then, by combining
Lemmas 13 and 15 we bound the second term in (44) with probability 1 − 4 δ .
Thus, the final result follows from the union bound ( Appendix F.10).
Constraint Violation : The violation term can be decomposed as the following
[Equation]
where K = T/H .
Note that the third term is equivalent to ( T/H ) κ .
The second term can be bounded by Azuma Hoeffding inequality ˜ O ( √ THι ) with probability 1 − δ .
We now describe how to bound the first term.
Note from Lemma
[Equation]
Now, using η =
[Equation]
√
KH 2 , Y ≤ ξ , Lemma 13 and 15 we obtain with probability 1 − 4 δ
[Equation]
Now, from the result of convex optimization (Corollary 1) we obtain that with probability 1 − 3 δ ,
K
[Equation]
Thus, we obtain bound for the second term.
The final result is obtained from the bounds of each individual term in (51).
F. P RELIMINARY R ESULTS
Lemma 16.
Under Assumption 2, for any fixed policy π , let w πh be the corresponding weights such that Q πj,h = ⟨ ϕ ( x, a ) , w πj,h ⟩ , for j ∈ { r, g } , then we have for all h ∈
[ H ] ,
[Equation]
Proof.
From the linearity of the action-value function, we have
[Equation]
where w πj,h
= θ j,h + (cid:82) S V πj,
h +1 ( x ′ ) dµ ( x ′ ) .
Now, || θ j,h ||≤ √ d , and || (cid:82) S V πj,
h +1 ( x ′ )
dµ h
( x ′ )
||≤ H √ d .
Thus, the result follows.
Lemma 17.
For any ( k, h ) , the weight w kj,h satisfies
[Equation] [Equation]
Proof.
For any vector v ∈ R
d we have
[Equation]
here π h,k ( ·| x ) is the Soft-max policy.
Note that Q kj,h +1 ( x, a ) ≤ H for any ( x, a ) .
Hence, from (57) we have
[Equation] [Equation]
We use Observation 1 to bound the second term.
Note that || w kj,h || =
max v : || v || =1 | v T w kj,h | .
Hence, the result follows.
F. P ROOF OF L EMMA
Proof.
We prove the result for ⋄ = r .
The proof for ⋄ = g is similar.
For any x , and h ∈ [ H ] ,
[Equation]
( x
[Equation]
Hence, | V πr,h ( x ) − ( H + h − 1) J πr |≤ sp( v πr ) since | v πr |≤ 1 2sp( v ∗ r ) .
F. P ROOF OF L EMMA
We first state and prove the following result which is similar to the one proved in Ding et al. (2021).
Lemma 18.
For Y ∈ [0 , ξ ] ,
K
[Equation]
Proof.
[Equation]
Summing over k , we obtain
[Equation]
Since Y 1 =
[Equation]
we have the result.
Now, we prove Lemma
[Equation]
Proof.
Note that
Y
K
[Equation]
) where the first inequality follows from Lemma 18, and the second inequality follows from the fact that V π ∗ g, 1 ( x k 1 ) ≥ Hb − κ .
Hence, the result simply follows from the above inequality.
F. P ROOF OF L EMMA
Proof.
We only prove for ⋄ =
r , the proof for ⋄ =
g is similar.
Note that Q πr,h ( x, a ) = ⟨ ϕ ( x, a ) , w πr,h ⟩ = r h ( x, a ) + P V πr,h +1 ( x, a ) .
We can write w πr,h
= (Λ k ) − 1 Λ k w πr,h .
Now, Λ k
= λ
I + (cid:80) k − 1 τ =1 (cid:80)
h ′ ϕ
( x τ h ′ , a τ h ′ )
ϕ ( x τ h ′ , a τ h ′ )
T (See the algorithm 1, line 3).
Hence, w πr,h = (Λ k ) − 1 ( λ I + (cid:80) k − 1 τ =1 (cid:80) h ′ ϕ
( x τh ′ , a τh ′ )
ϕ
( x τh ′ , a τh ′ ) T ) w πr,h
Finally, from the definition of w πr,h , ϕ ( x τh , a τh ) T w πr,h = r h ( x τh , a τh ) + P V r,h +
( x τh , a τh ) (from Lemma 4).
Hence, w πr,h = (Λ k ) − 1 λ I w πr,h + (Λ k ) − 1 (cid:80) k − 1 τ =1 (cid:80) h ′ ϕ
( x τh ′ , a τh ′ )
( r h ( x τh ′ , a τh ′ )
+ P V πr,h +1 ( x τh ′ , a τh ′ ))
Hence, we have w kr,h − w πr,h
=
(Λ k ) −k − 1 (cid:88) τ =
[Equation]
Now, we bound each term in the right hand side of expression in (62).
We call those terms as q 1 , q 2 , and q 3 respectively.
First, note that
[Equation]
Second, from Lemma 10, for the event in E , we have
[Equation] [Equation]
where χ = log(2( C 1 + 1) log( |A| ) ξdT/p ) .
Third, ⟨ ϕ ( x, a ) , q
[Equation]
= ⟨ ϕ ( x, a ) , (Λ k ) −
[Equation] [Equation]
The last term in (65) can be bounded as the following
[Equation]
) , λ (Λ k ) − 1 (cid:90) ( V kr,h +1 − V πr,h +1 )( x ′ ) dµ ( x ′ ) ⟩|≤
2 H √ dλ (cid:113) ϕ
( x, a ) T (Λ k ) − 1 ϕ
( x, a ) (66) since || (cid:82) ( V kr,h
+1 − V πr,h +1 )
( x ′ ) dµ ( x ′ )
||
2 ≤ 2 H √ d as || µ ( S ) ||≤ √ d .
The first term in (65) is equal to
[Equation]
Note that ⟨ ϕ ( x, a ) , w kr,h ⟩ − Q πr,h ( x, a ) = ⟨ ϕ ( x, a ) , w kr,h − w πr,h ⟩ = ⟨ ϕ ( x, a ) , q 1 + q 2 + q 3 ⟩ .
Since λ = 1 , we have from (63), (64,(66), and (67) |⟨
ϕ
( x, a ) , w kr,h ⟩ − Q πr,h ( x, a )
[Equation]
x
[Equation]
√
[Equation]
ϕ ( x
[Equation]
T (Λ
[Equation]
− 1 ϕ ( x, a ) (68) for some constant C 3 which is independent of C 1 .
Finally, note that
[Equation] [Equation]
where ι = log(2 log( |A| ) ξdT/p )
The last inequality follows from the fact that ι ∈ [log 2 , ∞ ) as | A |≥ 2 , and C 3 is independent of C 1 .
Hence, we can always pick C 3 (cid:112) log 2 + log( C 1 + 1) ≤
C 1 √ log 4 which satisfies (69) for all values of ι ∈ [log 2 , ∞ ) .
F. P ROOF OF L EMMA
Proof.
From Lemma 11, we have w.p. 1 − 2 δ ,
Q
[Equation]
F. P ROOF OF L EMMA
First, we state and prove a supporting result which bounds the value functions corresponding to the greedy policy and the soft-max policy at a given step.
We show that this gap can be controlled by the parameter α .
Lemma 19.
[Equation]
where
Definition 4.
¯ V kh ( · ) = max a [
[Equation]
, a
[Equation]
Y
[Equation]
· , a )] . ¯ V kh ( · ) is the value function corresponds to the greedy-policy with respect to the composite Q -function.
Proof.
Note that
[Equation]
where
[Equation]
Denote a x = arg max a [ Q kr,h ( x, a ) + Y k Q kg,h ( x, a )]
Now, recall from Definition 4 that ¯ V kh ( x ) =
[ Q kr,h ( x, a x ) + Y k Q kg,h ( x, a x )] .
Then,
[Equation]
where the last inequality follows from Proposition 1 in Pan et al.
We are now ready to show Lemma 13.
Proof.
We prove the lemma by Induction.
First, we prove for the step H .
Note that Q k ⋄ ,H +1 =
= Q π ⋄ ,H +1 .
Under the event in E as described in Lemma 10 and from Lemma 11, we have for j = r, g ,
[Equation]
x
[Equation] [Equation]
Hence, for any ( x, a ) ,
[Equation]
Hence, from the definition of ¯ V kh ,
[Equation]
for any policy π .
Thus, it also holds for π ∗ .
Hence, from Lemma 19, we have
[Equation]
Now, suppose that it is true till the step h
+
and consider the step h .
Since, it is true till step h + 1 , thus, for any policy π ,
[Equation]
From (48) in Lemma 12 and the above result, we have for any ( x, a )
[Equation]
Hence,
[Equation]
Now, again from Lemma 19, we have ¯ V kh ( x ) − V kh ( x ) ≤ log( |A| )
[Equation]
Thus,
[Equation]
π,Y k
[Equation]
[Equation]
Now, since it is true for any policy π , it will be true for π ∗ .
From the definition of V π,Y k , we have
[Equation]
Hence, the result follows by summing over K and considering h = 1 .
F. P ROOF OF L EMMA
Proof.
By Lemma 11, for any x, h, a, k
[Equation]
Thus,
[Equation]
Since V k ⋄ ,h ( x ) = (cid:80) a π h,k ( a | x ) Q k ⋄ ,h ( x, a ) and V π k ⋄ ,h ( x ) = (cid:80) a π h,k ( a | x ) Q π k ⋄ ,h ( x, a ) where π h,k ( a |· ) = S OFT -M AX aα ( Q kr,h + Y k Q kg,h ) ∀ a .
Thus, from (81),
V k ⋄ ,
[Equation]
k ⋄ ,
[Equation]
(cid:88)
a
[Equation]
] ≤
[Equation]
a
[Equation]
a
[Equation]
] +
[Equation]
Thus, from (82), we have
[Equation]
Hence
[Equation]
by iterating recursively
[Equation]
we have
[Equation]
The result follows.
F. P ROOF OF L EMMA
Proof.
Note from Lemma 14, we have with probability 1 − 2 δ ,
K (cid:88) k
=
[Equation]
We, now
[Equation]
bound the individual terms.
First, we show that the first term corresponds to a Martingale difference.
For any ( k, h ) ∈ [ K ] × [ H ] , we define F kh, 1 as σ -algebra generated by the state-action sequences, reward
[Equation]
and constraint values, { ( x τi , a τi ) } ( τ,i ) ∈
[ k − 1] ×
[ H ] ∪ { ( x ki , a ki ) } i ∈
[ h ] .
Similarly, we define the F kh, 2 as the σ -algebra generated by { ( x τi , a τi ) } ( τ,i ) ∈
[ k − 1] ×
[ H ] ∪ { ( x ki , a ki ) } i ∈
[ h ] ∪ { x kh +1 } .
x
kH +1 is a null state for any k ∈ [ K ] .
A filtration is a sequence of σ -algebras {F kh,m } ( k,h,m ) ∈
[ K ] × [ H ] × [2] in terms of time index
[Equation]
which holds that F kh,m ⊂ F k ′ h ′ ,m ′ for any t ≤ t ′ .
Note from the definitions in (50) that D k ⋄ ,h, 1 ∈ F kh, 1 and D k ⋄ ,h, 2 ∈ F kh, 2 .
Thus, for any ( k, h ) ∈
[ K ] × [ H ] ,
[Equation]
Notice that t ( k, 0 , 2) =
t ( k − 1 , H, 2) =
( H − 1) k .
Clearly, F k 0 , 2 = F k − 1 H, 2 for any k ≥ 2 .
Let F 10 ,be empty.
We define a Martingale sequence
[Equation] [Equation]
where t ( k, h, m ) =
( k − 1) H + 2( h − 1) + m is the time index.
Clearly, this martingale is adopted to the filtration {F kh,m } ( k,h,m ) ∈
[ K ] × [ H ] × [2] , and particularly
[Equation]
Thus, M K ⋄ ,H, 2 is a Martingale difference satisfying | M K ⋄ ,H, 2 |≤ 4 H since | D k ⋄ ,h, 1 | , | D k ⋄ ,h, 2 |≤ 2 H From the Azuma-Hoeffding inequality, we have
[Equation]
With probability 1 − 2 δ at least for any ⋄ = r, g ,
[Equation] [Equation]
Now, we bound the second term.
Note that the minimum eigen value of Λ kh is at least λ = 1 for all ( k, h ) ∈
[ K ] ×
[ H ] .
By Lemma 20,
[Equation]
Moreover, note that || Λ K
+1 ||
= || (cid:80) kτ =1 (cid:80) h ϕ kh
( ϕ kh ) T
+ λ I ||≤ λ + T , hence,
[Equation]
Now, by Cauchy-Schwartz inequality, we have
[Equation] [Equation]
Note that β = C 1 dH √ ι .
Thus, we have with probability 1 − 4 δ ,
[Equation] [Equation]
Hence, the result follows.
F. P ROOF OF T HEOREM
Note from Lemma 9, Lemma 13, and Lemma 15, we have w.p. 1 − 4 δ ,
[Equation]
[Equation]
Replacing Y with 0 in (96), we have
[Equation]
By noting that η = ξ √
KH 2 , and α = log |A|
[Equation]
) , we have
[Equation]
97) where the last equality follows from the fact that KH = T .
Now, observe the third term in (45).
Note from Azuma-Hoeffding inequality (since E [ V r, 1 ( π k )( x k 1 ) − (cid:80) kHt =(
[Equation]
1) H +1 ( r ( x h , a h ))]
= 0 with respect to the Filtration F k − 1 ).
Hence, with probability 1 − δ
[Equation]
Now, using Lemma 3, (97), and (98), in (44) we obtain with probability 1 − 5 δ ,
[Equation] [Equation]
Since H = T 1 / 4 /d 3 / 4 , we obtain with probability 1 − 5 δ ,
[Equation]
Now, we bound Violation.
By noting that η =
[Equation]
√
[Equation]
1 +
[Equation]
+
[Equation]
) , we have from (96),
[Equation]
From the convex optimization result (Corollary 1), we obtain
[Equation]
Also note from Azuma-Hoeffding inequality, w.p. 1 − δ
[Equation]
Hence, combining (102) and (103), we obtain w.p. 1 − 5 δ ,
K (cid:88)
[Equation]
Violation( T ) ≤ ( T/
[Equation]
κ +
[Equation]
where the last equality follows from the fact that H = T 1 / 4 /d 3 / 4 .
F D IFFERENCES IN A NALYSIS FOR A LGORITHMS 1 AND
Note that we also need to show that log ϵ -covering number for v function class is also log( T ) in
Algorithm 1.
However, since Algorithm 1 and Algorithm 2 are different, the analysis is also different.
In particular, in Algorithm 2 we consider a primal-dual type algorithm, hence, the policy is based on the combined state-action value function.
Thus, we need to show that the log ϵ -covering number for individual value function scales at most with log( T ) even though the policy is based on the composite
Q -function.
On the other hand, in Algorithm 1, we directly search for the policy which solves a constrained optimization problem.
In Algorithm 2 the policy is itself a function of Q ⋄ and dual variable Y k .
Thus, we obtain the result only using the property of the function class of Q ⋄ and the boundedness of Y k .
On the other hand a separate policy function class is defined for Algorithm 1 and we obtain the result by exploiting of the smoothness property of the function class.
Thus, our analysis would work for any policy class with smoothness property (Appendix E.6) unlike Algorithm 2.
We need that the optimal policy should also belong to Π in Algorithm 1.
In contrast, in Algorithm 2 we consider an unconstrained version of the episodic CMDP setup.
Carefully crafted Soft-max policy enables us to bound the gap with the optimal policy in the episodic setup.
Hence, we do not need to search over a feasible policy space unlike in Algorithm 1.
F. S UPPORTING R ESULTS
The following result is shown in Abbasi-Yadkori et al.
(2011) and in Lemma D.2 in Jin et al.
Lemma 20.
Let { ϕ t } t ≥ 0 be a sequence in ℜ d satisfying sup t ≥
|| ϕ t ||≤ 1 .
For any t ≥ 0 , we define Λ K = Λ 0 + (cid:80)
Kk =1 (cid:80)
h ϕ kh ( ϕ kh ) T .
Then if the smallest eigen value of Λ 0 be at least 1 , we have
[Equation]
Theorem 4.
[Concentration of Self-Normalized Process Abbasi-Yadkori et al.
(2011)] Let { ϵ t } ∞ t = be a real-valued stochastic process with corresponding filtration {F t } ∞ t =0 .
Let ϵ t |F t − 1 be a zero mean and σ sub-Gaussian, i.e., E [ ϵ t |F t − 1 ] = 0 , and
[Equation]
∈ ℜ ,
[Equation]
Let { ϕ t } ∞ t =1 be a ℜ d -valued Stochastic process where ϕ t ∈ F t − 1 .
Assume Λ 0 ∈ ℜ d × d is a positivedefine matrix, let, t = kH , Λ t = Λ 0 + (cid:80) kj =1 (cid:80)
h ϕ jh ( ϕ jh ) T .
Then for any δ > 0 with probability at least 1 − δ , we have
[Equation]
The next result characterizes the covering number of an Euclidean ball (Lemma 5.2 in Vershynin (2010)).
Lemma 21.
Let { x t } ∞ t =1 be a stochastic process on state space X with corresponding filtration {F t } , { ϕ t } be a R d -values stochastic process where ϕ t ∈ F t − 1 and || ϕ t ||≤ 1 , Λ t =
λ I + (cid:80) t − 1 s =1 ϕ
s
ϕ
Ts , and V be an arbitrary set of functions defined on X with N ϵ be its ϵ -covering number with respect to dist ( V, V ′ )
=
sup x | V ( x ) − V ′
( x ) | for some fixed ϵ > 0 .
Then for any δ > 0 with probability 1 − δ for any t , v ∈ V so that sup x | v ( x ) |≤ H , we have
[Equation]
Observation 1.
Let Λ k
= (cid:80) k − 1 τ =1 (cid:80) h ′ ϕ τh ′ ( ϕ τh ′ )
T
+ λ I , where λ > 0 , and ϕ τh ′ ∈ R d , then
[Equation]
1 (cid:88)
[Equation]
Proof.
We have (cid:80) k − 1 τ =1 (cid:80) h ′ ( ϕ τh ′ )
T (Λ k ) − 1 ϕ τh ′
=
(cid:80)
k − 1 τ =1 (cid:80) h ′ tr(( ϕ τh ′ )
T (Λ k ) − 1 ϕ τh ′ ) .
Further, (cid:80) k − 1 τ =1 (cid:80) h ′ tr(( ϕ τh ′ )
T (Λ k ) − 1 ϕ τh ′ )
=
tr((Λ k ) − 1 (cid:80)
k − 1 τ =1 (cid:80)
h ′ ϕ
τh ′ ( ϕ τh ′ ) T ) .
Given the eigen value decomposition, (cid:80) k − 1 τ =1 (cid:80) h ′ ϕ τh ′ ( ϕ τh ′ )
T =
U diag( λ 1 , . . .
, λ d ) U T .
Thus, Λ k = U diag( λ 1 + λ, . . .
, λ d + λ ) U T .
Hence, tr((Λ k ) − 1 (cid:80) k − 1 τ =1 (cid:80)
h ′ ϕ
τh ′ ( ϕ τh ′ ) T ) =
(cid:80) dl
=1 λ l / ( λ l + d ) ≤ d .
Lemma 22.
[Covering Number of Euclidean Ball]
For any ϵ > 0 , the ϵ -covering number of the
Euclidean ball in R d with radius R is upper bounded by (1 + 2 R/ϵ ) d .
We have used the following result from the optimization which is proved in Lemma 9 in Ding et al.
Lemma 23.
Let Y ∗ be the optimal dual variable, and C ≥ 2 Y ∗ , then, if
[Equation]
(109) then
[Equation]
Corollary 1.
Let Y ∗ be the optimal dual variable, and C ≥ 2 Y ∗ , if
[Equation]
then
[Equation]
Proof.
Let δ k
= V π ∗ r, 1 ( x k
[Equation]
− V π k r, 1 ( x k 1 ) + C ( b − V π k g, 1 ( x k 1 ) , then from Lemma 23,
[Equation]
By summing over k , we obtain
[Equation]
where the last inequality follows from the fact that (cid:80) k
δ k ≤ δ .
G P ROOF OF R ESULTS IN S ECTION
Organization : We, first, provide a proof sketch of Theorem 3 in (Appendix G.1).
Subsequently, we provide a detailed proof.
We show that how we can achieve zero constraint violation via tweaking the algorithm in Appendix G.5.
Notations for this Section : As shown in Appendix A of Wei et al.
(2021a), we assume that || ϕ ( x, a ) || 2 ≤ 1 .
Without loss of generality, we assume that we represent ϕ ( x, a ) T e 1 = 1 for some vector e 1 for every ( x, a ) .
|| µ ( S ) || 2 ≤ √ d , || θ j || 2 ≤ √ d for j = r, g .
Also, for every policy π , q π ( x, a ) can be written as ϕ ( x, a ) T w π .
|| w π ||≤ 6 t mix √ d (by Lemma 34).
We denote q kj ( x, a ) as ( w kj ) T ϕ ( x, a ) .
G. P ROOF S KETCH OF T HEOREM
Regret can be decomposed as the following
[Equation]
In the following, we bound each of the two terms at the right hand side.
Unlike the unconstrained setup Wei et al.
(2021a), here the decision depends on both q kr and q kg , thus, the analysis will also be different.
In particular, in order to bound the first term, we bound (cid:80) T/Bk = [Equation] [Equation]
∗ r [Equation] π k r ]+ BY ( b − J π k g ) .
Specifically, since J ∗ g ≥ b , we obtain for any Y ∈ [0 , ξ ]
[Equation]
Note that since the above expression is true for any Y ∈ [0 , ξ ] , we can recover the bound on regret by setting Y = 0 .
For T 1 , we employ the value-difference lemma.
This step may seem similar to the one considered for the unconstrained version (Wei et al., 2021a).
However, there are two important differences.
We have an additional constraint term scaled by the dual variable Y k which also changes over time.
Second, we provide high probability bound instead of expectation bound in Wei et al.
(2021a).
We utilize the one-step decent result of OMD from Lemma 3.3 in Cai et al.
(2020), Azume Hoeffdin inequality, and the boundedness of || w k ⋄
|| using Assumption 6 to bound the term.
The final result is shown in Lemma 24.
Lemma 24.
With probability 1 − δ ,
[Equation]
Proof.
See Appendix G.1.1.
Note that in the unconstrained setup the term T 2 does not arise.
For T 2 , we rely on the fact E k
( ˆ J k − J π k g ) are close since the way we collected samples ˆ J k is close to an unbiased estimator for J π k g .
Using the fact that | Y − Y k |≤ ξ , we bound T 2 as the following
Lemma 25.
With probability
[Equation]
Proof.
See Appendix G.1.2.
In order to bound T 3 , we utilize the result from the dual-domain analysis (Corollary 3).
We show that the upper bound of T 3 ≤ ξ √ TB .
Hence, we have with probability 1 − 2 δ ,
[Equation]
In order to bound the second term in (115), we need to show that policy between two epochs change slowly.
Here, we use the value of the hyper-parameter α , and the upper bound of the dual variable to bound the term
Lemma 26.
With probability 1 − 2 δ ,
[Equation]
Proof.
See Appendix G.
Hence, combining the above, setting Y = 0 and replacing (118) in (115) we obtain with probability 1 −
[Equation]
Now, replacing the value of α , and B , we obtain
[Equation]
Constraint Violation : Similar to the regret step, we decompose the violation in the following form
[Equation]
The second term can be upper bounded in the similar way as the second term in (115) with probability 1 − 2 δ .
For the first term, we observe
[Equation]
Now, from (118), we have with probability 1 − 2 δ
[Equation]
Finally, by replacing B and applying Corollary 4, we obtain with probability 1 − 2 δ
[Equation]
Hence, the result follows.
G.1. P ROOF OF L EMMA
We, first, state the following results which will be useful for proving Lemma 24.
Lemma 27.
Using the value difference lemma (Lemma 15) in Wei et al.
(2020), we have
[Equation]
where ν π ∗ ( x ) is the stationary distribution corresponding to the stationary policy π ∗ .
Also note that (cid:80) a ( π ∗ ( a | x ) − π k ( a | x ))
N
( J π k r + Y k J π k g )
=
since J π k ⋄ is constant.
From Lemma 15 in Wei et al.
(2021a)
Lemma 28.
Let E [ ·| τ k,m ] denote the expectation conditioned on ( x τ k,m , a τ k,m ) and all the history before τ k, [Equation] , then
[Equation]
Thus, R k,m and G k,m are close to the unbiased estimator of q π k r + NJ π k r and q π k g + NK π k g respectively.
Also from Lemma 16 in Wei et al.
(2021a)
Lemma 29.
For ⋄ = r, g ,
[Equation]
[Equation]
where E k is the expectation conditioned on all history before epoch k .
Now, we are ready to prove Lemma 24.
Proof From Lemma 27, We express T 1 in (115) as
[Equation]
Bounding T 4 : The bound of T 4 is similar to the unconstrained case Wei et al. (2021a). The only difference is that since we have extra term corresponding to Y k q kg , there is an additional scaling factor of (1 + ξ ) . The details are in the following–
Note that q π k ⋄ ( x, a ) = ( w π k ⋄ ) T ϕ ( x, a ) , J π k ⋄ = J π k ⋄ e T 1 ϕ ( x, a ) ,and q k ⋄ ( x, a ) = ( w k ⋄ ) T ϕ ( x, a ) .
From Lemma 29, we obtain that
[Equation]
T Hence,
[Equation]
≤
[Equation]
+
[Equation]
T
Hence,
[Equation]
Bounding T 5 : Since we are providing high probability bound instead unlike the expected regret bound in Wei et al.
(2021a), we need to bound T 5 whereas such a term did not arise in Wei et al.
(2021a). T 5 is Martingale difference with respect to the Filtration F k − 1 which contains all the histories till epoch k . | q kr
[Equation] x, a ) + Y k q kg ( x, a ) |≤ (1 + ξ ) N/σ (From Lemma 31).
Hence, from Azuma-Hoeffding inequality, we obtain w.p. 1 − δ ,
[Equation]
Thus, multiplying the above with B , we obtain
[Equation]
[Equation]
with probability 1 − δ .
Bounding T 6 : In the unconstrained case Wei et al. (2021a) bounds E
[ (cid:80) k (cid:80) a ( π ∗ ( a | x ) − π k ( a | x ))
q k r ( x, a )]
(Lemma 19 in Wei et al.
(2021a)).
T 6 may seem similar to the above expression, however, there are subtle differences.
First, we do not have expectation since we need to consider high-probability bound.
Second, we have additional Y k q kg term where Y k is changing over k .
Thus, we need to use different techniques compared to Wei et al.
(2021a).
In particular, we rely on the analysis of OMD and one-step decent result from Lemma 3.3 in Cai et al.
(2020) to bound T 6 .
We bound T 6 in Lemma 30.
We use the result from Lemma 30 to bound T 6 –
[Equation] [Equation]
since KB = T .
Hence, we have with probability 1 − δ ,
B
K (cid:88) k
= (cid:88)
a
[Equation]
Since α = min { σ (1 + ξ ) √ Tt mix , σ [Equation] + ξ
[Equation]
} , N = 8 t
[Equation] log( T ) , B = 32 N log( dT ) σ − 1 , then, B
[Equation]
where the last inequality follows from the dominated convergence theorem and (cid:82) X dν π ∗ ( x ) = 1 .
Hence, we have with probability 1 − δ ,
[Equation]
G.1. P ROOF OF L EMMA
Proof.
We can decompose T 2 as (cid:88)
[Equation]
From Lemma 33, | E k ( ˆ J k ) − J π k |≤ 2 T 7 .
Also note that ( Y − Y k )( ˆ J k − E k
( ˆ J k )) is a Martingale sequence where ˆ J k ≤ 1 .
Further, | Y − Y k |≤ ξ .
Thus, we have with probability 1 − δ ,
K (cid:88)
[Equation]
Since K = T/B , hence, with probability 1 − δ
[Equation]
From Corollary 3 and η = ξ (cid:112) T/B , we have
[Equation]
(136) where recall that T 3 is the third term in (115).
G. P ROOF OF L EMMA
Proof.
Note that
[Equation]
k
[Equation]
[Table]
Note that | v π 1 ( x 11 ) |≤ O ( t mix ) , and | v π K ( x KKB ) |≤ O ( t mix ) , thus, we have
[Equation]
The first term is a Martingale sequence.
Hence, with prob.
1 − δ , we have
[Equation]
For the second term, since | π k ( a | x ) − π k − [Equation] a | x [Equation]
|≤ O
[Equation]
α [Equation] N [Equation]
σ
[Equation]
π
[Equation]
a | x
[Equation]
) (by Lemma 32), then, by Lemma 7 in Wei et al. (2020), we have
[Equation]
[Equation]
The third term is again a Martingale sequence.
Hence, we have with probability 1 − δ ,
[Equation]
Hence, we have with probability 1 − 2 δ ,
[Equation]
Now, using the value of α , we obtain with probability 1 − 2 δ ,
[Equation]
[Equation]
G. C ONSTRAINT V IOLATION B OUND
From (118), and the value of α , N , and B , we have with probability 1 − 2 δ (cid:88)
[Equation]
Now, from Corollary
[Equation]
On the other hand similar to Lemma 26, we can also show that
Corollary 2.
With prob.
1 − 2 δ ,
[Equation]
Hence, we obtain with probability 1 − 4 δ ,
[Equation]
G. S UPPORTING R ESULTS
Similar to Lemma 18, we can show the following
Corollary 3.
For any Y ∈ [0 , ξ ]
[Equation] [Equation]
Lemma 30.
[Equation]
Proof.
Now, from the composition of the algorithm (12), we obtain
[Equation]
From Lemma 3.3 in Cai et al.
[Equation]
Now, from Lemma 31, ||
q k − 1 r
[Equation]
k q k
[Equation] 1 g [Equation] Now, by summing from k = 2 to K + 1 , and shifting k − 1 to k , we obtain from (149),
K (cid:88)
[Equation]
=
[Equation]
[Equation]
By Pinsker’s inequality
[Equation]
On the other hand, since π 1 is uniformly random, thus,
[Equation]
Hence, from (150),
K (cid:88)
[Equation]
Lemma 31.
For any x , and a
[Equation]
Proof.
[Equation]
Note that from the construction of the parameter,
[Equation]
Hence, we have
[Equation]
Lemma 32.
For any a, x
[Equation]
Proof.
[Equation]
Note that α max b | ϕ ( x, b ) T ( w k − 1 r
[Equation]
≤ 1 as long as α ≤ σ/ (24(1 + ξ ) N ) .
Combining with the fact that exp(2 x ) ≤ 1 + 8 x for x ∈ [0 , 1] .
[Equation]
Hence, the result follows.
Lemma 33.
For any
[Equation]
Proof.
Let P π k
( x t , x 1 ) denote the probability that the learner reaches the state x kt at time t within epoch k when the epoch starts with the state x k 1 .
Hence,
[Equation]
where δ x 1 ( x ) is 1 when x = x 1 , and 0 otherwise.
[Equation]
where the second last inequality follows from Assumption 4.
The last inequality follows from the definition of N .
Similar to Corollary 1, we can show the following
Corollary 4.
For C ≥ 2 Y ∗ ,
[Equation]
161) then [Equation]
[Equation]
Finally, we state some results which are proved in Wei et al.
(2021a)
(Lemma 6 and 14, respectively).
Lemma 34.
For any stationary policy π , and x | v π ( x ) |≤ 4 t mix , and | q π ( x, a ) |≤ 6 t mix for any ( x, a ) .
Lemma 35.
[Equation]
G. Z ERO C ONSTRAINT V IOLATION
We show that by considering a ϵ -tighter optimization problem, and by carefully choosing ϵ , one can obtain zero constraint violation while maintaining the same order on the regret.
First,we introduce some notations.
Consider the ϵ -tighter optimization problem
[Equation]
subject to
[Equation]
where in the constraint b + ϵ replaces b .
If ϵ ≤ γ/ 2 , then Slater’s condition is satisfied as ¯ π would still be strictly feasible for the ϵ -tighter CMDP (Assumption 3).
The optimal dual variable is bounded by Y ∗ ≤ 2 /γ for this tighter optimization problem.
Since we need ξ ≥ 2 Y ∗ , we make ξ = 4 /γ .
We denote the optimal policy of the ϵ -tighter optimization problem as π ϵ, ∗ .
We denote J π ϵ, ∗ as the optimal constant gain of the tighter optimization problem (164).
Now, we are ready to state the result for the set of Assumptions in Section 4.
Lemma 36.
In Algorithm 3 replace b with b + ϵ , and ξ with 4 /γ , then under Assumptions 3,2, 4, and 5, with probability 1 − 4 δ
[Equation]
where ϵ = min { C 6 (2(1 + ξ ) σξ (cid:112)
Tt 3 mix log( dT ) 2 /δ
T ) , γ/ 2 } for some absolute constant C 6 .
Note that when ˜ O (
[Equation]
γ/ 2 , then violation becomes 0 .
Hence, for some constant
C 6 , we obtain violation 0 when C 6 (2(1 + ξ ) σξ [Equation]
≤ γ/ 2 .
Also note that by plugging in the value of ϵ , the upper bound on regret is still ˜ O ( √ T ) .
Hence, even when for large enough T (still finite), the violation is 0 , while the regret bound is still ˜ O ( √ T ) .
Proof : First, we prove the upper bound on regret.
The regret can be decomposed as the following -
[Equation]
The first term can be bounded with the help of the following lemma (the proof for finite state is in
Wei et al.
and for episodic setup is in Wei et al.
(2021b), we provide the proof at the end of this section)–
Lemma 37.
If each stationary policy π induces a stationary state-action distribution ν π ( x, a ) , every stationary policy satisfies (4), and π ϵ, ∗ is the optimal solution of (164), then
[Equation]
Under uniform mixture assumption (Assumption 4, the conditions of the Lemma can be verified.
Since the tighter optimization problem is also CMDP, we note that the second term in the right hand side of (166) is essentially the regret of the tighter CMDP.
Hence, from Theorem 3 and Lemma 37 we obtain the expression of the regret bound in Lemma 36.
Constraint Violation : Again applying Theorem 3 to the tighter optimization problem (164), we obtain
[Equation]
Thus,
[Equation]
Thus, the result follows.
Now, we show that zero violation can be attained under the set of Assumptions in Theorem 2 and one additional assumption:
Assumption 6.
Every stationary policy π satisfies (4), and induces a state-action stationary distribution ν π ( x, a ) .
For uniform mixture assumption (Assumption 4), one can show that Assumption 6 holds.
Note that
Wei et al.
(2022) which studies the infinite horizon average reward CMDP for the finite state space setup assumes the above assumption inherently in their work.
We are now ready to state and prove the main result.
Lemma 38.
Replace b with b + ϵ in Algorithm 2, κ = max { sp( v ¯ πr , sp( v π ϵ, ∗ r ) } , and ξ = 4 /γ , then, under Assumptions 3,2, 1, and 6,
[Equation]
where ϵ = min { C 7 ((1 + κ )( dT ) 3 / 4 )
ι
T , γ/ 2 } for absolute constant C 7 , and ι = log(2 log( |A| ) dT/δ ) .
Note that when ˜ O ((1 + κ )( dT ) 3 / 4 )
T ≤ γ/ 2 , the violation becomes 0 .
Hence, there exists some constant C 7 such that when C [Equation]
[Equation]
T ≤ γ/ 2 , then the violation becomes 0 .
Hence, for large enough T (Still finite), the violation indeed becomes 0 .
On the other hand, by plugging in the value of ϵ , we note that the regret bound is still ˜ O (( dT ) 3 / 4 ) (in fact it is ˜ O
((1+ κ +sp( v ∗ r ))( dT ) 3 / 4 ) .
Proof.
First, we prove the upper bound on regret.
The regret can be decomposed as the following -
[Equation]
The first term is bounded from Lemma 37 since we assume that every stationary policy satisfies (4).
Hence,
[Equation]
Since the tighter optimization problem is also CMDP, we note that the second term in the right hand side of (169) is essentially the regret of the tighter CMDP.
Hence, from Theorem 2 and Lemma 37 we obtain the expression of the regret bound in Lemma 38.
Constraint Violation : Again applying Theorem 2 to the tighter optimization problem (164), we obtain
[Equation]
Thus,
T
[Equation]
We can also obtain that zero violation for Theorem 1 in a similar way while maintaining same order of regret by picking ϵ
= O (1 / √ T ) in the tighter CMDP.
Here, we show the proof of Lemma 37.
Proof.
Let ν π ( x, a ) be the stationary state-action occupation distribution corresponding to the stationary policy π .
According to Assumption, for every policy induces a stationary distribution.
Hence,
J [Equation] and J [Equation]
Consider the state-action distribution ν ϵ ( · , · ) =
(1 − [Equation]
[Equation] ) ν π ∗ ( · , · )+ [Equation] [Equation]
¯ π ( · , · ) .
There exists a policy ˜ π which induces the state-action distribution ν ϵ Altman (1999).
Now,
[Equation]
Thus, such a state-action occupation measure is feasible for the tighter optimization problem (164).
Further,
[Equation]
Since the state-action measure ν ϵ ( x, a ) is feasible, then J π ϵ, ∗ r ≥ (cid:82) x,a r ( x, a ) dν ϵ ( x, a ) .
Hence, from (172), we have
[Equation]
Hence, the result follows.
H E XPERIMENTS
We evaluate Algorithm 3 on a similar model described in Chen et al.
; Singh et al.
(2020)– a wireless node is continuously transmitting packets.
The node consists of a queue where the maximum capacity is 9 .
At time t , the node needs to choose a transmission power a t ∈ { 0 . 1 , 0 . 9 } as action.
Higher transmission power results in higher probability of successful transmission.
The number of packets arriving at the node at time t is given by Y t .
We assume Y t ∈ { 0 , 1 , 2 , 3 } , and the corresponding probabilities are (0 . 65 , 0 . 2 , 0 . 1 , 0 . 05) .
The channel reliability is 0 .
9 , i.e., each attempted transmission has success probability of 0 . 9 .
The dynamics of the queue is the following
Q t +1 = min { 9 , max { Q t + Y t − D t , 0 }} where D t is 1 with probability a t p r , and 0 otherwise.
The goal is to maintain a short queue length with small transmission power.
At time t , the node gets a reward 1 − a t .
Reward decreases as a t increases.
On the other hand, it gets a utility if the state of the queue is low.
Specially, the utility is g ( Q t ) = 1 − 0 .
1 Q t .
We seek to ensure that the average utility is kept at least 0 .
7 , i.e., the goal is to keep average queue length less than or equal to 3 .
Compared to Chen et al.
(2022), we use a different utility function.
First, we ensure that the utility is bounded between 0 and 1 .
Further, Chen et al.
(2022) considers the constraint as a cost function where we consider it as utility function.
Hence, we need to provide higher utility for lower state and vice versa unlike in Chen et al.
Further, our algorithm is model-free unlike in Chen et al.
Note that the setup can be represented in a tabular form.
Since linear MDP contains tabular form, the feature space representation becomes simple, in particular, ϕ ( s, a )
=
e s,a where e s,a is 1 for the state-action pair ( s, a ) , and 0 otherwise.
The dimension of the feature space is |S||A| .
5 T 100.
A v e r age
R eg r e t     5 T 10 0.
A v e r age
V i o l a t i on We run Algorithm 3 for 5 × 10 5 time steps.
The parameter we used are the followings: B = 300 ,
N = 10 , α = 1 / √ T , η = 10 / (cid:112) T/B .
Note that we do not know t mix or σ , yet, we achieve sub-linear regret and a violation which approaches zero.
Hence, our algorithm can work well even if we do not know the MDP dependent parameters.
In Fig. 1 we plot the average regret and average violation as a function of T for ϵ = 0 .
01 (the tightness for constrained introduced in Remark 1).
As predicted by our theory, our algorithm achieves sub-linear regret and sub-linear violation as both the average regret and average violation decrease to 0 as T increases.
Initially, the regret [-is negative, [Figure] eg r e t    4 K 
 C on s t r a i n t-]decreases fast since we start with an infeasible policy (violation increases, Fig. 3).
Subsequently, the regret increases as our algorithm seeks to reduce the violation by making the dual variable high. The average violation decreases rapidly (Fig. 2). Eventually, the regret again decreases when T ≈ 1 × 10 5 as our algorithm finds the dual variable which balances the regret and violation.
In this regime, the average violation slowly approaches 0 as T grows.
In Fig. 2 we plot the average regret and average violation as a function of T for ϵ = 0 . 1 . Again it is apparent that the regret and violation grow sub-linearly as the average regret and average violation decrease to 0 . Since ϵ is high, from Fig.
3 we observe that the violation indeed eventually becomes 0 . From Fig. 3 we observe that the violation starts decreasing after T ≈ 0 .
5 × 10 5 as opposed to T ≈ 10 5 for smaller value of ϵ . The variation of regret is similar to Fig.
1 as the regret first decreases rapidly and then increases till T ≈ 0 . 5 × 10 5 . The regret eventually decreases steadily as T increases further.
Note that the regret starts to decrease much earlier (at T ≈ 0 .
5 × 10 5 ) in this scenario compared to the scenario where ϵ is smaller. Intuitively, since ϵ is higher, the dual variable increases at a higher rate which helps in finding the feasible solution at a faster rate. The regret then starts decreasing as once we find the dual variable which balances between the regret and violation.
We would like to point out the difference with Chen et al.
The violation in Chen et al.
(2022) oscillates, and never approaches 0 .
On the other hand, in our evaluation, we observe that the violation eventually approaches 0 .
This is due to the fact that we have a learning rate η (which is O (1 / √ T ) ) in the dual-update step whereas in Chen et al.
(2022) the learning rate is 1 , hence, in Chen et al.
the dual variable oscillates more compared to our approach.
Further, Chen et al.
(2022) only guarantees constant violation, whereas we can achieve zero violation. [Figure] A v g . V i o
l a t
i
[Figure]
[Figure]