Semi-Infinitely Constrained Markov Decision Processes
Anonymous Author(s)
AffiliationAddress email
Abstract
We propose a generalization of constrained Markov decision processes (CMDPs)that we call the semi-infinitely constrained Markov decision process (SICMDP).
Particularly, we consider a continuum of constraints instead of a finite number ofconstraints as in the case of ordinary CMDPs.
We also devise a reinforcementlearning algorithm for SICMDPs that we call SI-CRL.
We first transform thereinforcement learning problem into a linear semi-infinitely programming (LSIP)problem and then use the dual exchange method in the LSIP literature to solve it.
To the best of our knowledge, we are the first to apply tools from semi-infinitelyprogramming (SIP) to solve constrained reinforcement learning problems.
Wepresent theoretical analysis for SI-CRL, identifying its sample complexity anditeration complexity.
We also conduct extensive numerical examples to illustratethe SICMDP model and validate the SI-CRL algorithm
Introduction
Reinforcement learning has achieved great success in areas such as Game-playing [34, 39], robotics[24], recommender systems [44], etc.
However, due to safety concerns or physical limitations, insome real-world reinforcement learning problems, we must consider additional constraints that mayinfluence the optimal policy and the learning process [15].
A standard framework to handle suchcases is the constrained Markov Decision Process (CMDP) [5].
Within the CMDP framework, theagent has to maximize the expected cumulative reward while obeying a finite number of constraints,which are usually in the form of expected cumulative cost criteria.
However, we are sometimes concerned with the problem with a continuum of constraints.
Forexample, the constraints we meet might be time-evolving or subject to uncertain parameters, whichcannot be formulated as an ordinary CMDP (see Examples 3.1 and 3.2).
In this paper we wouldstudy a generalized CMDP to address the above problem.
Because the constraints are not onlyinfinite-number but also lie in a continuous set, the generalization is not trivial.
Fortunately, we findthat we can borrow the idea behind linear semi-infinite programming (LSIP)
[33, 16] to deal withthe semi-infinite constraints.
Accordingly, we propose semi-infinitely constrained Markov decisionprocesses (SICMDPs) as a novel complement to the ordinary CMDP framework.
We also present a so-called SI-CRL reinforcement learning algorithm to solve SICMDPs.
Themain challenge is that we need to deal with a continuum of constraints, thus reinforcement learning algorithms for ordinary CMDPs do not work anymore.
We tackle this difficulty by first transforming the reinforcement learning problem to an equivalent LSIP problem, which can then be solved usingthe dual exchange methods in the LSIP literature [23, 32].
As far as we know, we are the first tointroduce tools from semi-infinitely programming (SIP) into the reinforcement learning community for solving constrained reinforcement learning problems.
Furthermore, we give theoretical analysis for SI-CRL.
We decompose the error of SI-CRL intotwo parts: the statistical error from approximating the true SICMDP with an offline dataset and the optimization error due to the fact that the solution of the LSIP problem obtained by the dualexchange method is inexact.
On the statistical side, we show that the sample complexity of SI-CRLis (cid:101) O (cid:16) | S | 2 | A | 2 ϵ 2 (1 − γ ) 3 (cid:17) if the offline dataset is generated by a generative model, and (cid:101) O (cid:16) | S
||
A | ν min ϵ 2 (1 − γ ) 3 (cid:17) ifthe dataset is generated by a probability measure ν as considered in [11].
Here (cid:101)
O means that alllogarithm terms are discarded.
On the optimization side, we show that the iteration complexity of
SI-CRL is O (cid:18)(cid:110) diam( Y ) L (cid:112) | S | 2 | A | d/ [(1 − γ )
ϵ ] (cid:111) d (cid:19) .
We perform a set of numerical experiments to illustrate the SICMDP model and validate the SI-CRLalgorithm.
We consider two numerical examples: toy SICMDP and discharge of sewage.
In theexample of toy SICMDP, we show the efficiency of the SI-CRL algorithm and validate the establishedtheoretical bounds.
In the example of discharge of sewage, we further show the advantage of the
SICMDP framework over the CMDP baseline obtained by naive discretization in modeling realisticdecision-making problems.
Related Work
The constrained Markov decision processes (CMDPs) have been extensively applied in areas likerobotics [31], communication and networks, [27, 35] and finance [1].
For a detailed treatment of
CMDPs one may refer to [5].
A number of reinforcement learning algorithms for CMDPs areproposed, which include Lagrangian methods [4], actor-critic methods [2, 37], policy gradientmethods [42], etc.
There are also works focusing on theoretical aspects of CMDPs.
Wu et al.[41], Amani et al.
[6] studied the online regret bound of the bandit case.
Wachi and Sui [40], Zhengand Ratliff [45] considered the case where the reward and cost are random but the transition dynamicsare known.
And Efroni et al.
[14], Amani et al.
[7], HasanzadeZonuzy et al.
[21] considered thecase where the transition dynamics are unknown and need to be estimated.
Our SI-CRL algorithmuses a similar strategy as in [14] in the sense that they all use the optimistic method to transform thereinforcement learning problem into a linear (semi-infinitely) programming problem, which resolvesthe feasibility issue and makes the theoretical analysis easier as well.
However, our work and [14]are very different at the technical level: 1) Our theoretical guarantees are in the form of samplecomplexity bounds, while the results in (Efroni et.
al 2020) are in the form of online regret bounds;the proof techniques are quite different.
2) Efroni et al.
[14] considered the episodic MDPs, while weconsider the infinite-horizon case.
The origination of semi-infinitely programming (SIP) can date back to [33].
From then on, SIP hasbeen widely used in quantum physics [10], signal processing [29, 30], finance [13], environmentscience, and engineering [22].
One important class of SIP problems is called linear semi-infinitelyprogramming (LSIP).
Goberna and López [17] provided a thorough survey about LSIP theory.
Variousnumerical methods are proposed to solve LSIP problems, including discretization methods [9, 13],exchange methods [23, 43], and local reduction methods [20, 12].
Unlike LP, most LSIP problemscannot be solved exactly and all-purpose LSIP solvers do not exist.
In SI-CRL, we choose to use thedual exchange method in [23] to solve the LSIP problem therein for its conceptual simplicity as wellas concrete theoretical guarantees.
The SICMDP Model
A semi-infinitely constrained MDP (SICMDP) is defined by a tuple M = ⟨ S, A, Y, P, r, c, u, µ, γ ⟩ .
Here S, A, P, r, µ, γ are defined in a similar manner as in common infinite-horizon discounted MDPs.
Specifically, S and A are the finite sets of states and actions, respectively.
P is the transition dynamicsand P ( s ′ | s, a ) represents the probability of transitioning to state s ′ when playing action a at state s .
And r : S × A → [0 , 1] is the reward function, µ is the fixed initial distribution, and γ is the discountfactor.
Y is the set of constrains, which we define as a compact set in R d , and diam( Y ) < ∞ denotesits diameter.
In addition, c : Y × S × A → [0 , 1] is used to denote a continuum of cost functionsand the value for constraints (bounds that must be satisfied) is determined by function u :
Y → R .
Note that when Y is finite, we get an ordinary constrained MDP, which is indeed a special case of
SICMDP.
The general SICMDP problem is to find a stationary policy π : S → ∆( A ) , where ∆( A ) is the setof probability measure supported on A , to maximize the value function while complying with acontinuum of constraints.
In other words, we consider the following optimization problem:
[Equation]
where V π ( µ ) :
=
E ( (cid:80) ∞ t =0 γ t r ( s t , a t ) | s 0 ∼ µ ) and C πy ( µ ) :
=
E ( (cid:80) ∞ t =
γ t c y ( s t , a t ) | s 0 ∼ µ ) .
Let us see two concrete examples of SICMDPs.
Example 3.1 (Spatial-temporal Constraints) .
Consider an ordinary CMDP problem with a singleconstraint:
[Equation]
In some cases the constraint would be spatial-temporal, i.e., the cost function c ( s, a ) and the valuefor constraints u are no longer constant function and would change with time τ ∈ [0 , T ] or locationd ∈ D ⊂ R 3 .
Then we should use the SICMDP model with Y =
[0 , T ] or Y = D rather than theordinary CMDP framework to model such problems:
[Equation]
Load Balancing : Suppose a RL agent needs to balance the load between multiple cell sites usingsome policy π .
The objective is to minimize the cost V π ( µ ) and the constraint is that at every place din the region D the cumulative communication capacity C πd ( µ ) is above some threshold u d .
Example 3.2 (Constraints with Uncertainty) .
Again we consider a problem like Problem (1).
Inmany application scenarios the cost function c ( s, a ) is handcrafted and the construction of c ( s, a )is not guaranteed to be correct.
Hence it may be helpful to include an additional parameter
ϵ ∈
Erepresenting our uncertainty in the construction of the cost function c ( s, a ) as well as the value ofconstraints u .
Even if the constraint is not handcrafted and has clear physical meaning, it may stillsubject to uncertain parameters ϵ ∈ E that cannot be observed in advance.
Therefore, we should usethe SICMDP model with Y =
E rather than the ordinary CMDP framework to model such problems:
[Equation]
Underwater Drone : Suppose an underwater drone needs to maximize V π ( µ ) to accomplish sometasks.
When the unknown environment feature (salinity, temperature, ocean current, etc,) is ϵ ∈ E ,for state-action pair ( s, a ) the energy consumption is c
ϵ
( s, a ) , and the constraint is that total energyconsumption C πϵ ( µ ) cannot be larger than its battery capacity u ϵ .
Remark 3.3 .
An alternative approach to solving problems such as Examples 3.1 and 3.2 is to naivelydiscretize the constraint set Y , and then the discretized problem can be fit into the conventional
CMDP framework.
The problem of this naive method is that the prior knowledge, i.e., the constraintfunction is continuous w.r.t.
y , would be lost, which makes the method extremely inefficient.
In
Section 6.2 we demonstrate this issue via a numerical example.
When an SICMDP M is known to us, we may do the planning by solving a linear semi-infinite programming (LSIP) problem.
Denote the occupancy measure on S × A introduced by policy π as q π ∈ ∆( S × A ) .
Then we have
[Equation]
Problem (M) can be reformulated as the following LSIP problem:
[Equation]
Therefore, when M is already known the optimal policy π ∗ can be found by solving Problem (4).
And we always assume such a policy π ∗ exists.
Assumption 3.4.
Problem (M) is feasible with an optimal solution π ∗ , or equivalently, Problem (4)is feasible with an optimal solution q ∗ .
The SI-CRL Algorithm
In this section we present an offline reinforcement learning algorithm called SI-CRL for SICMDPs.
In a high-level point of view, our algorithm is a semi-infinite version of the algorithms proposed in[21, 14].
In the first stage, SI-CRL takes an offline dataset { ( s i , a i , s ′ i ) | i = 1 , 2 , . . .
, m } as input andgenerate an empirical estimate (cid:98) P of the true transition dynamic P .
Then the algorithm constructsa confidence set (the optimistic set) according to (cid:98) P that would cover the true SICMDP with highprobability. Then for each policy π we would only view its return as the largest possible return in SICMDPs in the confidence set. This method is also called the optimistic approach.
In the secondstage, the optimistic policy ˜ π is found using a LSIP algorithm.
It can be shown that the resultingpolicy ˜ π is guaranteed to be nearly optimal, and the theoretical analysis can be found in Section 5.
Now we give a more detailed description of SI-CRL.
First, the empirical estimate (cid:98) P is calculated as:(cid:98) P ( s ′ | s, a ) :
= n ( s,a,s ′ ) max(1 ,n ( s,a )) , where n ( s, a, s ′ ) :
= (cid:80) mi =1 1 {
s i =
s, a i = a, s ′ i =
s ′ } and n ( s, a ) =(cid:80) s ′
n ( s, a, s ′ ) .
The reason why we do not directly plug (cid:98) P into Problem (4) and solve the resulting
LSIP problem is due to the fact that there is no guarantee that the LSIP problem w.r.t.
(cid:98) P is feasible.
To address this issue, we construct an optimistic set M δ such that with high probability the true
SICMDP M lies in M δ .
In particular, M δ is defined via the empirical Bernstein’s bound and the
Hoeffding’s bound [26]:
[Equation]
whered
[Equation]
The next step is to solve the optimistic planning problem:
[Equation]
where the superscript M ′ denotes that the expectation is taken w.r.t.
SICMDP M ′ .
Theorem 4.1.
Suppose n ≥ 3 .
With probability at least 1 − 2 | S | 2 | A | δ , we have that M ∈ M δ , and
Problem (5) is feasible.
The proof is given in the appendix.
Note that the optimization variables include both M ′ and π ,and LSIP reformulations like Problem (4) would no longer be possible.
Instead, we shall introduce the state-action-state occupancy measure z
( s, a, s ′ ) .
In particular, assuming z P,π ( s, a, s ′ ) :
=
P ( s ′ | s, a ) q π
( s, a ) , we have P ( s ′ | s, a ) =
z P,π ( s,a,s ′
[Equation]
148 Problem (5) can be reformulated as the following extended LSIP problem:
[Equation]
However, compared to LP problems, LSIP problems are typically harder to solve and there are noall-purpose LSIP solvers.
Here, we choose the simple yet effective dual exchange methods [23, 32]to solve Problem 6.
The SI-CRL algorithm can be summarized in Algorithm 1.
Algorithm
Input: state space S , action space A , dataset { ( s i , a i , s ′ i ) | i = 1 , 2 , ..., m } , reward function r , a continuum of cost function c , value for constraints u , discount factor γ for each ( s, a, s ′ )
tuple do
Set (cid:98) P ( s ′ | s, a ) :
= (cid:80) mi
[Equation]
[Equation]
mi =1 1 { s i =
s,a i = a } ) end for
Initialize Y
= { y 0 } for i = 1 to T do
Use a LP solver to solve a finite version of Problem (6) by only considering constraints in Yand store the solution as z i
Find y i ≈ argmax y ∈ Y (cid:80) s,a,s ′
z i ( s, a, s ′ )
c y
( s, a )
− u y
if (cid:80) s,a,s ′
z ( s, a, s ′ )
c
y i ( s, a ) − u y i ≤ 0 then
Set z T =
z
i
BREAK end if
Add y i to Yend for for each ( s, a ) pair do
Set ˆ π (
[Equation]
=
[Equation]
,s
[Equation]
) end for
RETURN ˆ π
Theoretical Analysis
We give PAC-type bounds for SI-CRL under two different settings.
The error of SI-CRL is decomposed into two parts: the statistical error from approximating Problem (M) with Problem (5) andthe optimization error from the fact that the solution of (5) obtained by dual exchange method isinexact.
On the statistical side, our goal is to determine that how many samples are required to make
SI-CRL an ( ϵ, δ ) -optimal 1 when Problem (5) can be solved exactly, i.e., we want to find the samplecomplexity of SI-CRL.
We show that the sample complexity of SI-CRL is (cid:101) O (cid:16) | S | 2 | A | 2 ϵ 2 (1 − γ ) 3 (cid:17) if the datasetwe use is generated by a generative model, and (cid:101) O (cid:16) | S ||
A | ν min ϵ 2 (1 − γ ) 3 (cid:17) if the dataset we use is generatedby a probability measure ν defined on the space S
× A and P ( ·| s, a ) as considered in [11].
Here (cid:101) Omeans that all logarithm terms are discarded, and ν min :=
min ν ( s,a ) >
ν
( s, a ) .
It can be noted thatthe order of our sample complexity bound increases by a factor of | S
||
A | compared to that of ordinarydiscounted MDP [8].
On the optimization side, we show that if the inner maximization problemw.r.t.
y can be solved exactly, the dual exchange method would produce an ϵ -optimal solutions 2 whenthe number of iterations T
= O (cid:18)(cid:104) diam( Y ) L (cid:112) | S | 2 | A | d/ϵ (cid:105) d (cid:19) , where L is the Lipschitz constantdefined in Assumption 5.3.
We will present our theoretical analysis in more details in the followingpart of this section.
5.1 Notation and Preliminaries
Given a stationary policy π , we define the value function V π ( s ) =
E ( (cid:80) ∞ t =0 γ t r ( s t , a t ) | s 0 = s ) ,
V π =
( V π ( s 1 ) , . . .
, V π ( s | S | ))
⊤ ∈ R | S | .
Thus we have V π ( µ ) =
µ ⊤ V π .
Similarly, we define theexpected cost C πy ( s ) =
E ( (cid:80) ∞ [Equation] = [Equation]
γ
[Equation] c y ( s [Equation] , a
[Equation]
) |
[Equation]
0 =
[Equation]
[Equation]
= (
[Equation]
s
[Equation]
, C πy ( s | S | ))
⊤ ∈ R | S | , thus
C πy ( µ ) = µ ⊤ C πy .
And π ∗ denotes the optimal policy.
Suppose ˜ π, (cid:102)
M are the solution of Problem( 5 ) and (cid:102)
M = ⟨ S, A, Y, (cid:101) P, r, c, u, µ, γ ⟩ .
For a given stationary policy π , we use (cid:101) V π ( s ) , (cid:101) V π , (cid:101) V π ( µ ) ,(cid:101) C π y ( s ) , (cid:101) C π y , (cid:101) C π y ( µ ) , ˜ q π ( s, a ) , to represent the value function, expected cost, occupancy measure of
SICMDP (cid:102) M , respectively.
We say an offline dataset { ( s i , a
i , s ′ i ) | i = 1 , 2 , . . .
, m } to be generated bya generative model if we sample according to P ( ·| s, a ) for each ( s, a ) -pair n = m/ | S ||
A | times and record the results in the dataset.
We say an offline dataset to be generated by probability measure νand P ( ·| s, a ) if ( s i , a i ) i.i.d.
∼ ν and s ′ i ∼ P ( ·| s i , a i ) .
An ( ϵ, δ ) -optimal policy is defined as follows.
Definition 5.1.
An RL algorithm is called ( ϵ, δ ) -optimal for ϵ, δ > 0 if with probability at least 1 − δ it can return a policy π such that
[Equation]
An ϵ -optimal solution of Problem (5) is defined as
Definition 5.2.
A stationary policy ˆ π is called an ϵ -optimal solution of Problem (5) for ϵ >
if
[Equation]
hold simultaneously.
Unless otherwise specified, we assume that ∀ ( s, a ) ∈ S × A , c y ( s, a ) is L -Lipschitz in y w.r.t.
∥·∥ 2 .
We also assume that u y is L -Lipschitz in y w.r.t.
∥·∥ 2 .
The assumptions can be formally stated as:
Assumption 5.3.
c y ( s, a ) and u y are Lipschitz in y w.r.t.
∥·∥ 2 , i.e., ∃ L > 0 s.t. ∀
y, y ′ ∈ Y, ( s, a ) ∈
S × A, | c y ( s, a ) − c y ′
( s, a )
| ≤ L ∥ y − y ′
∥ 2 , | u y − u y ′
| ≤ L ∥ y − y ′
∥ 2 .
The Lipschitz assumption is usually necessary when dealing with a semi-infinitely constrainedproblem [36, 23].
And this assumption is indeed quite mild because Y is a compact set.
5.2 Sample Complexity of SI-CRL
We consider the case where the offline dataset we use is generated by a generative model.
First weconsider a restricted setting as in [26] where for each ( s, a ) -pair in the true SICMDP there are at mosttwo possible next-states and provide the sample complexity bound.
Then we will drop Assumption5.4 using the same strategy as in [26] and derive the sample complexity bound of the general case.
The proof can be found in the appendix.
Assumption 5.4.
The true unknown SICMDP M satisfies P ( s ′ | s, a ) =
for all but two s ′ ∈
Sdenoted as sa + and sa − ∈ S .
Theorem 5.5.
Suppose Assumption 5.4 holds, and the dataset we use is generated by a generative model with m/ | S
||
A | = n > max (cid:110) 36log4 /δ
(1 − γ ) 2 , 4log4 /δ
(1 − γ ) 3 (cid:111) .
Then with probability 1 − 2 | S | 2 | A | δ , we have that
[Equation]
Theorem 5.6.
Suppose Assumption 5.4 holds, the dataset we use is generated by a generativemodel and Problem 5 can be solved exactly.
Then when m = O (cid:18) | S ||
A | log ( 8 | S | 2 | A |
[Equation]
ϵ 2 ( [Equation] − γ
[Equation]
(cid:19) , SI-CRL is( ϵ, δ ) -optimal.
Theorem 5.7.
Suppose the dataset we use is generated by a generative model and Problem 5 can besolved exactly.
Then when m = O (cid:18) | S | 2 | A | 2 (log | S | ) 3 log ( 8 | S | 4 | A | 3 /δ )
ϵ 2 (1 − γ )(cid:19) , a modification of SI-CRL is( ϵ, δ ) -optimal.
Remark 5.8 .
Our proof strategy is similar to [26].
However, to get a (cid:101) O
((1 − γ ) − 3 ) bound [26] uses atedious recursion argument.
We greatly simplify the proof and achieve improvements in log terms(by a factor of (log( | S | ϵ (1 − γ ) )) 2 ) using sharper bounds on local variances of MDPs developed in [3].
Remark 5.9 .
Although Assumption 5.4 seems quite restrictive, we argue that it is necessary toestablish sharp sample complexity bound, as shown in [26].
Specifically, without this assumption the“quasi-Bernstein bound” (Lemma B.4) will not hold, thus we may not be able to get the (cid:101) O
((1 − γ ) − 3 )bound.
Remark 5.10 .
It can be noted that our sample complexity bound does not rely on the constraint set Y .
This is because we consider the setting where r and c y are known deterministic functions and theonly source of randomness comes from estimating the unknown transition dynamic using an offlinedataset.
Now we generalize our results to the case where the offline dataset is generated by a probabilitymeasure.
The proof can be found in the appendix.
Theorem 5.11.
Suppose the dataset we use is generated by probability measure ν and Problem 5 canbe solved exactly.
Then when m = O (cid:18) | S ||
A | (log | S | ) 3 log ( 8 | S | 4 | A | 3 /δ )
ν min ϵ 2 (1 − γ )(cid:19) , a modification of SI-CRL is( ϵ, δ ) -optimal.
Remark 5.12 .
Here "a modification of SI-CRL" stands for the following procedure: first we transformthe original SICMDP to a new SICMDP satisfying Assumption 5.4, then we run SI-CRL to solve thenew SICMDP.
One may refer to the proof in Appendix B for more details.
5.3 Iteration Complexity of SI-CRL
We give the iteration complexity of SI-CRL, i.e., how many iterations are required to output anϵ -optimal solution of Problem (5) when the inner-loop problem can be solved exactly.
Our results is acorollary of Theorem 4 in [23].
Theorem 5.13.
If the inner-loop maximization problem in SI-CRL can be exactly solved, then
SI-CRL will output an ϵ -optimal solution of Problem (5) if the number of iterations T =
O (cid:18) (cid:110) diam( Y ) L (cid:112) | S | 2 | A | d/ [(1 − γ )
ϵ ] (cid:111) d (cid:19) .
Numerical Experiments
We design two numerical examples: toy SICMDP and discharge of sewage.
By a set of numericalexperiments, we illustrate the SICMDP model and validate the efficacy of the SI-CRL algorithmas well as the correctness of our theoretical results.
We highlight that in the example of dischargeof sewage we find that the SICMDP framework greatly outperforms the CMDP baseline obtainedby discretizing the original problem in modeling realistic reinforcement learning problems.
Weimplement our methods with Python and LP problems are solved using a full-featured universityversion of Gurobi [19].
Details of our implementation can be found in the appendix.
All theexperiments are run on a workstation with 8 CPUs and no GPU.
6.1 Toy SICMDP
We consider a most simple SICMDP with | S | = 2 , | A | = 2 and Y =
Its MDP part is specifiedin Figure 1, where p ∈ (0 . 5 , 1) and τ ≪ 1 is a small positive number.
For each γ ∈ (0 , 1) , we design
Lipschitz c y and u y such that the optimal policy takes a 0 with probability 0.9 and 0.5 on s 0 and s 1 ,respectively.
For details of the construction of Toy SICMDP, one may refer to the appendix.
To makeour numerical results more reliable, we repeat all experiments in this subsection for 30 times andreport the average results.
First, we would like to check the efficacy of the SI-CRL algorithm.
We set
[Equation]
244 T sufficiently large such that the algorithm is guaranteed to converge.
Then we gradually increasem , the size of the dataset, and see whether SI-CRL can recover the pre-defined optimal policy.
Theresults are shown in Figure 3.
It can be noticed that as m gets larger, the error term converges tozero, showing that our SI-CRL algorithm may effectively solve reinforcement learning problems for
SICMDPs.
Second, we would like to validate the theoretical results in Section 5.
Specifically, we investigate the sample complexity of SI-CRL for a fixed ( ϵ, δ ) (See Definition 5.1) when γ and ν minvary.
T is set to be sufficiently large as in the previous experiment.
We present the results in Figure 3.
The logarithm of sample complexity vs. the transformed parameter of interest is approximately linearwith slope 1, which indicates our sample complexity bounds are correct and tight.
[Figure]
6.2 Discharge of Sewage
To demonstrate the power of the SICMDP model and the SI-CRL algorithm, we consider a morerealistic and complex problem adapted from [18].
Assume there are | S | sewage outfalls in a region[0 , 1] d , with d = 2 or 3 , and at each time point only one single outfall is active.
The active outfallwould cause pollution in nearby areas and the impact would decrease with Euclidean distance.
Weneed to figure out a strategy to switch between neighboring outfalls to avoid over-pollution at eachlocation of the region while minimizing the switching cost.
Clearly, this problem can be formulatedas a SICMDP model with Y =
[0 , 1] d and corresponding c y and u y .
For details of the constructionof the Discharge of Sewage, one may refer to the appendix.
In the following numerical experiments,we assume that an offline dataset generated by a generative model is available.
First, we numerically validate our theoretical bounds on sample complexity and iteration complexity.
In particular, we investigate the relationship between the sample complexity and iteration complexityof SI-CRL and the size of state space | S | .
Like the case in Toy SICMDP, we find the numericalresults fit well with our theoretical analysis.
We show the results in Figures 4.
As before, we run eachexperiment for 30 times and report the averaged results.
[Figure]
Second, we compare our method with a naive CMDP baseline 3.3, showing the advantage of SICMDPin modeling problems like Example 3.1, 3.2.
In the baseline method, we only consider the constraints on a grid of Y containing T baseline points, which allows us to model Discharge of Sewage as a standard
CMDP problem with T baseline constraints.
The CMDP problem is then solved by the algorithmproposed in [14].
We visualize the quality of solutions of our proposed method and baseline methodin Figure 5.
It can be found that when T = T baseline , the policy obtained by our proposed methods isof far better quality than the policy obtained by the baseline methods.
Larger number means more serious violation of constraints.
The red cross icons represent the
T =
T baseline = 9 check points selected by the algorithms.
An anti-intuitive phenomenon is that although in our method we need to deal with multiple LPproblems while in the baseline we only solve one single LP problem, our method is still moretime-efficient than the CMDP baseline.
Figure 7 indicates that our method takes less time to get asolution of given accuracy, which is evaluated by the error term sup y ∈ Y C
ˆ πy ( µ )
− u y .
The reason isthat in SI-CRL we solve LP problems with a dual simplex method, thus re-optimization after addinga new constraint is much faster than re-solving the LP problem from scratch[25].
And our methodneeds far fewer active constraints to attain the same accuracy as the baseline methods, see Figure 6.
[Figure]
when T and T baseline vary.
( δ = 0 . 005 | S | 2 |
A | , m sufficiently large)
[Figure]
[Figure]
Conclusion
We have studied a novel generalization of CMDP that we have called SICMDP.
In particular, we haveconsidered a continuum of constraints rather than a finite number of constraints.
We have deviseda reinforcement learning algorithm SI-CRL to solve SICMDP problems.
Furthermore, we havepresented theoretical analysis for SI-CRL, establishing the sample complexity bounds as well as theiteration complexity bounds.
We have also performed the extensive numerical experiments to showthe efficacy of our proposed method and its advantage over traditional CMDPs.
However, the SI-CRLalgorithm can only handle the tabular case, with a nice offline dataset avaliable.
We would study the
SICMDP beyond the tabular case and develop efficient algorithms in future works.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Assumptions 3.4, 5.3, 5.4.
(b) Did you include complete proofs of all theoretical results?
[Yes] All proofs can be found in Appendix A, B.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] Please seesupplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 6 and
Appendix D
(c) Did you report error bars (e
g., with respect to the random seed after running
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] All the experiments are run on aworkstation with 8 CPUs and no GPU.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A
Omitted Proofs in Section
Proof of Theorem 4.1.
By Lemma C.1,
[Equation]
By Lemma C.2,
[Equation]
Combining the two inequalities in a union bound, we have:
[Equation]
Again we apply the union bound argument to get:
[Equation]
Finally, Problem (5) is feasible as long as P ∈ M δ because of Assumption 3 . 4 .
B Omitted Proofs in Section 5
First we define some additional notations.
We use Q π ( s, a ) :
=
E ( (cid:80) ∞ t =0 γ t r ( s t , a )) to denote the state-action value function.
The local variance is defined as Var P ( V π )( s, a ) =
E s ′ ∼ P ( ·| s,a ) ( V π ( s ′ ) −
P ( ·| s, a ) V π ) 2 .
We view V π as vector of length | S | and Q π , r, Var P
( V π ) as vectors of length | S |·| A | .
We overload notation and let P also refer to a matrix of size ( | S |·| A | ) ×| S | , where the entry P ( s,a ) ,s ′ is equal to P ( s ′ | s, a ) .
We also define P π to be the transition matrix on state-action pairs induced by a stationary policy π , namely: P π
[Equation]
s,
[Equation]
[Equation] We use (cid:101) Q π ( s, a ) , Var (cid:101) P ( (cid:101) V π )( s, a ) , (cid:101) Q π , Var (cid:101) P ( (cid:101) V π ) , (cid:101) V π , (cid:101) σ 2 π , (cid:101) P, (cid:101) P π to denote the state-action valuefunction, local variance, vector of state-action value function, vector of local variance, vector of valuefunction, transition matrix, transition matrix on state-action pairs w.r.t.
SICMDP (cid:102) M , respectively.
Lemma B.1.
If Assumption 5.4 is true and M ∈ M δ , we have
[Equation]
Proof.
Given a stationary policy π , if Assumption 5.4 is true and M ∈ M δ ,
[Equation]
, which implies
[Equation]
1 − γ (cid:114)
[Equation]
Then we have
[Equation]
Lemma B.2.
Given a stationary policy π , when Assumption 5.4 is true and M ∈ M δ , we have
[Equation]
Proof.
For simplicity of notation, we drop the dependence on π .
By definition,
[Equation]
where ( · ) 2 means element-wise squares.
When Assumption 5.4 is true and M ∈ M δ , by Lemma B.1,
[Equation]
We also have
[Equation] [Equation]
B.1) .
Therefore, we can get
[Equation]
δ
[Equation]
Lemma B.3.
Let p, ˜ p, ˆ p ∈ [0 , 1] satisfy | p − ˆ p | ≤ min (cid:40)(cid:114) 2ˆ p (1 − ˆ p ) log 4 /δ n + 4 log 4 /δ n , (cid:114) log 2 /δ 2 n (cid:41) | ˜ p − ˆ p | ≤ min (cid:40)(cid:114) 2ˆ p (1 − ˆ p ) log 4 /δ n + 4 log 4 /δ n , (cid:114) log 2 /δ 2 n (cid:41) . Then | p − ˜ p | ≤ (cid:114) [Equation] p
[Equation] log [Equation] /δ n +(cid:18) log 4 /δ n
[Equation]
log 4 /
[Equation]
n
Proof.
Assume WLOG that ˆ p ≥ p .
Therefore,
[Equation]
n
[Equation]
Similarly, we have
[Equation]
n .
Thus we may complete the proof using triangular inequality.
Lemma B.4.
Given a stationary policy π , suppose Assumption 5.4 is true and M ∈ M δ , then
[Equation]
where ⪯ means every element of LHS is less than or equal to the its counterpart in RHS.
Proof.
Let p =
P ( sa + | s, a ) , ˜ p
= ˜ P
( sa + | s, a ) .
Applying Lemma B.3 yields
[Equation]
Assume WLOG that V π ( sa + ) ≥ V π ( sa − ) .
Therefore we have
[Equation]
Since p
[Equation]
Var P ( V π )
We may get | ( P ( ·| s, a ) −
[Equation]
P
[Equation]
·|
[Equation]
, a
[Equation]
≤
[Equation]
8Var
[Equation]
n (1 − γ ) , which completes the proof.
Lemma B.5.
Given a stationary policy π , suppose Assumption 5.4 is true and M ∈ M δ , then we have
[Equation]
Proof.
From Lemma C.3, Lemma C.4, Lemma C.5, Lemma B.4 and the fact that (cid:16) I − γ (cid:101) P π (cid:17) − 1 has positive entries, we know
[Equation]
n
[Equation]
(cid:17) − 1 (cid:112)
Var P ( V π ) (cid:13)(cid:13)(cid:13)(cid:13) ∞
+ (1 − γ )(cid:18) log 4 /δ n (cid:19) 3 /+
(1 − γ )(cid:18) log 4 /δ n (cid:19) ≤ (cid:114) 16 log 4 /δ n (cid:13)(cid:13)(cid:13) (cid:13) (cid:16) I − γ (cid:101) P π (cid:17) − 1 (cid:113)
Var (cid:101) P ( (cid:101)
V π )
(cid:13)(cid:13)(cid:13) (cid:13) ∞ + 4 √(1 − γ )(cid:18) log 4 /δ n (cid:19) 3 /+
(1 − γ )(cid:18) log 4 /δ n (cid:19) ≤ (1 − γ ) 3 /(cid:114) log 4 /δ n + 4 √(1 − γ )
(cid:18) log 4 /δ n (cid:19) 3 /+
(1 − γ )(cid:18) log 4 /δ n (cid:19) .
The proof is completed since (cid:13)(cid:13)(cid:13) V π − (cid:101)
V π (cid:13)(cid:13)(cid:13) ∞ ≤ (cid:13)(cid:13)(cid:13) Q π − (cid:101) Q
π (cid:13)(cid:13)(cid:13) ∞ by definitions.
Lemma B.6.
Suppose Assumption 5.4 is true and n > 6log4 /δ
(1 − γ ) 5 / 2 , then with probability at least 1 − 2 | S | 2 | A | δ , we have
[Equation]
Proof.
When Assumption 5.4 is true and M ∈ M δ , it follows from Lemma B.5 that
[Equation]
And by setting n > max (cid:110) 36log4 /δ
(1 − γ ) 2 , 4log4 /δ
(1 − γ ) 3 (cid:111)
we will get (cid:13)(cid:13) (cid:13) V π −
[Equation]
(cid:13) ∞ ≤
[Equation]
log
[Equation]
n
[Equation]
1 − γ ) 3 .
Similar arguments can be applied to bound (cid:13)(cid:13)(cid:13) C πy − (cid:101) C πy (cid:13)(cid:13)(cid:13) ∞ .
Since by Theorem 4.1 we have
[Equation]
≥ 1 − 2 | S | 2 | A | δ, the proof is completed.
Proof of Theorem 5.5.
By Lemma B.5, we know that with probability 1 − 2 | S | 2 | A | δ ,
[Equation]
[Equation]
[Equation]
Thus
[Equation]
Noting that ˜ V ˜ π ( µ ) ≥ ˜
V
π ∗ ( µ ) , we may get
[Equation]
Similarly, when
[Equation]
we may get
[Equation]
since ˜ C ˜ πy ( µ ) ≤ u y .
Proof of Theorem 5.6.
Theorem 5.6 is a direct corollary of Theorem 5.5.
Proof of Theorem 5.7.
The proof is nearly identical to the proof of Theorem 3 in [26].
The idea is toaugment each state/action pair of the original mdp with | S | − 2 states in the form of a binary tree aspictured in the diagram below.
The intention of the tree is to construct an SICMDP, ¯ M = ⟨ ¯ S, A, Y, ¯ P, ¯ r, ¯ c, u, µ, ¯ γ ⟩ that with appropriate transition probabilities is functionally equivalent to M while satisfying Assumption 5.4.
The rewards and costs in the added states are set to zero.
Since the tree has depth d = O (log 2 | S | ) , it now takes d time-steps in the augmented SICMDP to change states once in the original SICMDP.
Therefore we must also rescale the discount factor ¯ γ by setting ¯ γ < γ d .
Now we have
[Equation]
Then we complete the proof by applying results in Theorem 5.6.
Proof of Theorem 5.11.
By Theorem C.6, we have for any fixed ( s, a ) ∈ S ×
A
P ( n ( s, a ) <
mν min / 2) ≤ P ( n ( s, a ) < mν ( s, a ) / 2) ≤ e − mν ( s,a ) (cid:18) emν
( s, a ) emν
( s, a ) /(cid:19)
mν ( s,a )=
(cid:18)(cid:114) e (cid:19) − ν ( s,a ) m ≤ (cid:18)(cid:114) e (cid:19) − ν min m
Let m =
[Equation]
, we have P ( n ( s, a ) ≥ mν min )) ≥ 1 − δ/
| S ||
A | .
Therefore, with probability at least 1 − δ/ 2 , we can get
[Equation]
Then our problem is reduced to the case that the offline dataset is generated by generative models.
The proof is completed by using results in Theorem 5.7.
Proof of Theorem 5.13.
See Theorem 4 in [23].
C Auxiliary Lemmas
Lemma C.1 (Empirical Bernstein Inequality) .
Suppose n ≥ 3 , { X 1 , ..., X n } be n i.i.d.
random variables with values in [0 , 1] .
Let δ > 0 .
Then with probability at least 1 − δ we have
[Equation]
where V n ( X 1: n ) :=
[Equation]
denotes the empirical variance of the dataset{ X 1 , ..., X n } .
Proof.
See Theorem 11 in [28].
Lemma C.2 (Hoeffding’s Inequality) .
Suppose { X 1 , ..., X n } be n i.i.d.
random variables with values in [0 , 1] .
Let δ > 0 .
Then with probability at least 1 − δ we have
[Equation]
Proof.
See Theorem 2.2.6 in [38].
Lemma C.3.
For any policy π and transition probabilities P , (cid:101) P , we have that
[Equation]
Proof.
See Lemma 2 in [3].
Lemma C.4.
For any policy π , any transition probability P and any vector v ∈ R | S |·| A | , we have
[Equation]
Proof.
See Lemma 3 in [3].
Lemma C.5.
For any policy π and any transition probability P , we have
[Equation]
where √· is defined as the element
-wise square root
Proof.
See Lemma 4 in [3].
Theorem C.6 (Chernoff’s Inequality) .
Let X i be independent Bernoulli random variables with parameter p i .
Consider their sum S N = (cid:80) Ni
=1 X i and denote its mean by µ = E S N .
Then, for any t < µ , we have
[Equation]
Proof.
See [38].
D Details of Numerical Experiments
We will release codes of our algorithms and construction of corresponding environments on GitHubsoon.
D.1 Construction of Toy SICMDP
Without loss of generality, assume Y =
We split Y =
[0 , 3] to Y 1 =
[0 , 1] , Y 2 =
[1 , 2] and
Y 3 =
Intuitively, in Y 1 ( Y 3 ), we restrict the agent to take action a 0 in s 0 ( s 1 ) with the givenprobability from the target policy ˜ π .
(Recall, taking action a 0 is always better if we set aside theconstraints.)
We introduce Y 2 to obtain L -Lipschitz of c y and u y .
Assume f : [ − 0 .
→ Ris a continuous-differentiable even function, with unique maximum point 0 and zero point 0 .
5 ,we use f ( x ) =
(1 + cos(2 πx )) cos(2 πx ) in practice.
Let c y ( s 0 , a 0 ) =
f ( y − 0 . 5) for y ∈ Y 1 ,
c y ( s 1 , a 0 ) =
f ( y − 2 . 5) for y ∈ Y 3 , and c ≡ 0 otherwise.
For each γ ∈ (0 , 1) , we defineu 1 = C ˜ π 0 . 5 > 0 and u 1 = C ˜
π 2 .
5 > 0 .
Let u y ≡ u 1 in Y 1 , u y ≡ u 2 in Y 3 and we make linearinterpolate u y in Y 2 .
So far, we have constructed Lipschitz c y and u y .
The only active constraints are C π 0 .
5 ≤ u 1 and
C π 2 .
5 ≤ u 2 and the optimal policy is the known target policy ˜ π .
D.2 Construction of Discharge of Sewage
In the numerical experiments, we generate this environment randomly.
In each experiment, wesample the environments 30 times and report the average result.
Positions of sewage outfalls, transition dynamics and rewards are sampled uniformly on Y , probabilitysimplex and [0 , 1] respectively.
Assume f : [0 , + ∞ ) → [0 , + ∞ ) is a continuous-differentiable decreasing function, we use f ( x ) =1 1+ x 2 in practice.
Let c y
( s, a ) =
c y ( s ) =
f ( ∥ y − s ∥ 2 ) , where s also represents the position of thestate (outfall).
Given a target state-occupancy measure d which can be generated randomly or specified in advance,we define u y =
(1 + ∆) (cid:80) s
∈ S d ( s ) c y ( s ) , where ∆ is a small positive number.
The feasibility ofthe resulting SICMDP is not guaranteed even if ∆ = 0 , we reject those infeasible environments andre-sample.
The SICMDP would be nontrivial if we choose a suitable ∆ .
D.3 Implementation of SICMDP
The only thing we need to clarify is how to solve the maximization problem to generate new y iin i th iteration.
Since the cost function c y ( s, a ) can be non-convex in y , we choose to solve themaximization problem by brute force.
Specifically, we first create a grid of Y of size 10 5 , and thenfind the grid point with max objective.
This method works well since in the problems we consider Yis of low dimensions.