Semi-Infinitely Constrained Markov Decision Processes
Abstract
We propose a generalization of constrained Markov decision processes (CMDPs) that we call the semi-infinitely constrained Markov decision process (SICMDP).
Particularly, we consider a continuum of constraints instead of a finite number of constraints as in the case of ordinary CMDPs.
We also devise a reinforcement learning algorithm for SICMDPs that we call SI-CRL.
We first transform the reinforcement learning problem into a linear semi-infinitely programming (LSIP) problem and then use the dual exchange method in the LSIP literature to solve it.
To the best of our knowledge, we are the first to apply tools from semi-infinitely programming (SIP) to solve constrained reinforcement learning problems.
We present theoretical analysis for SI-CRL, identifying its sample complexity and iteration complexity.
We also conduct extensive numerical examples to illustrate the SICMDP model and validate the SI-CRL algorithm.
Introduction
Reinforcement learning has achieved great success in areas such as Game-playing [34, 39], robotics [24], recommender systems [44] etc.
However, due to safety concerns or physical limitations, in some real-world reinforcement learning problems, we must consider additional constraints that may influence the optimal policy and the learning process [15].
A standard framework to handle such cases is the constrained Markov Decision Process (CMDP) [5].
Within the CMDP framework, the agent has to maximize the expected cumulative reward while obeying a finite number of constraints, which are usually in the form of expected cumulative cost criteria.
However, we are sometimes concerned with the problem with a continuum of constraints.
For example, the constraints we meet might be time-evolving or subject to uncertain parameters, which cannot be formulated as an ordinary CMDP (see Examples 3.1 and 3.2).
In this paper we would study a generalized CMDP to address the above problem.
Because the constraints are not only infinite-number but also lie in a continuous set, the generalization is not trivial.
Fortunately, we find that we can borrow the idea behind linear semi-infinite programming (LSIP)
[33, 16] to deal with the semi-infinite constraints.
Accordingly, we propose semi-infinitely constrained Markov decision processes (SICMDPs) as a novel complement to the ordinary CMDP framework.
We also present a so-called SI-CRL reinforcement learning algorithm to solve SICMDPs.
The main challenge is that we need to deal with a continuum of constraints, thus reinforcement learning algorithms for ordinary CMDPs do not work anymore.
We tackle this difficulty by first transforming the reinforcement learning problem to an equivalent LSIP problem, which can then be solved using the dual exchange methods in the LSIP literature [23, 32].
As far as we know, we are the first to introduce tools from semi-infinitely programming (SIP) into the reinforcement learning community for solving constrained reinforcement learning problems.
Furthermore, we give theoretical analysis for SI-CRL.
We decompose the error of SI-CRL into two parts: the statistical error from approximating the true SICMDP with an offline dataset and the optimization error due to the fact that the solution of the LSIP problem obtained by the dual exchange method is inexact.
On the statistical side, we show that the sample complexity of SI-CRL is (cid:101) O (cid:16) | S | 2 | A | 2 ϵ 2 (1 − γ ) 3 (cid:17) if the offline dataset is generated by a generative model, and (cid:101) O (cid:16) | S ||
A | ν min ϵ 2 (1 − γ ) 3 (cid:17) if the dataset is generated by a probability measure ν as considered in [11].
Here (cid:101)
O means that all logarithm terms are discarded.
On the optimization side, we show that the iteration complexity of
SI-CRL is O (cid:18)(cid:110) diam( Y ) L (cid:112) | S | 2 | A | d/ [(1 − γ )
ϵ ] (cid:111) d (cid:19) .
We perform a set of numerical experiments to illustrate the SICMDP model and validate the SI-CRL algorithm.
We consider two numerical examples: toy SICMDP and discharge of sewage.
In the example of toy SICMDP, we show the efficiency of the SI-CRL algorithm and validate the established theoretical bounds.
In the example of discharge of sewage, we further show the advantage of the
SICMDP framework over the CMDP baseline obtained by naive discretization in modeling realistic decision-making problems.
Related Work
The constrained Markov decision processes (CMDPs) have been extensively applied in areas like robotics [31], communication and networks, [27, 35] and finance [1].
For a detailed treatment of
CMDPs one may refer to [5].
A number of reinforcement learning algorithms for CMDPs are proposed, which include Lagrangian methods [4], actor-critic methods [2, 37], policy gradient methods [42], etc.
There are also works focusing on theoretical aspects of CMDPs.
Wu et al.
[41], Amani et al.
[6] studied the online regret bound of the bandit case.
Wachi and Sui [40], Zheng and Ratliff [45] considered the case where the reward and cost are random but the transition dynamics are known.
And Efroni et al.
[14], Amani et al.
[7], HasanzadeZonuzy et al.
[21] considered the case where the transition dynamics are unknown and need to be estimated.
Our SI-CRL algorithm uses a similar strategy as in [14] in the sense that they all use the optimistic method to transform the reinforcement learning problem into a linear (semi-infinitely) programming problem, which resolves the feasibility issue and makes the theoretical analysis easier as well.
However, our work and [14] are very different at the technical level: 1)
Our theoretical guarantees are in the form of sample complexity bounds, while the results in (Efroni et.
al 2020) are in the form of online regret bounds; the proof techniques are quite different.
2) Efroni et al.
[14] considered the episodic MDPs, while we consider the infinite-horizon case.
The origination of semi-infinitely programming (SIP) can date back to [33].
From then on, SIP has been widely used in quantum physics [10], signal processing [29, 30], finance [13], environment science, and engineering [22].
One important class of SIP problems is called linear semi-infinitely programming (LSIP).
Goberna and López [17] provided a thorough survey about LSIP theory.
Various numerical methods are proposed to solve LSIP problems, including discretization methods [9, 13], exchange methods [23, 43], and local reduction methods [20, 12].
Unlike LP, most LSIP problems cannot be solved exactly and all-purpose LSIP solvers do not exist.
In SI-CRL, we choose to use the dual exchange method in [23] to solve the LSIP problem therein for its conceptual simplicity as well as concrete theoretical guarantees.
The SICMDP Model
A semi-infinitely constrained MDP (SICMDP) is defined by a tuple M = ⟨ S, A, Y, P, r, c, u, µ, γ ⟩ .
Here S, A, P, r, µ, γ are defined in a similar manner as in common infinite-horizon discounted MDPs.
Specifically, S and A are the finite sets of states and actions, respectively.
P is the transition dynamics and P ( s ′ | s, a ) represents the probability of transitioning to state s ′ when playing action a at state s .
And r : S × A → [0 , 1] is the reward function, µ is the fixed initial distribution, and γ is the discount factor.
Y is the set of constrains, which we define as a compact set in R d , and diam( Y ) <
∞ denotes its diameter.
In addition, c : Y × S × A → [0 , 1] is used to denote a continuum of cost functions and the value for constraints (bounds that must be satisfied) is determined by function u :
Y → R .
Note that when Y is finite, we get an ordinary constrained MDP, which is indeed a special case of
SICMDP.
The general SICMDP problem is to find a stationary policy π : S → ∆( A ) , where ∆( A ) is the set of probability measure supported on A , to maximize the value function while complying with a continuum of constraints.
In other words, we consider the following optimization problem:
[Equation]
where V π ( µ ) :
=
E ( (cid:80) ∞ t =0 γ t r ( s t , a t ) | s 0 ∼ µ ) and C πy ( µ ) :
=
E ( (cid:80) ∞ t =
γ t c y ( s t , a t ) | s 0 ∼ µ ) .
Let us see two concrete examples of SICMDPs.
Example 3.1 (Spatial-temporal Constraints) .
Consider an ordinary CMDP problem with a single constraint:
[Equation]
In some cases the constraint would be spatial-temporal, i.e., the cost function c ( s, a ) and the value for constraints u are no longer constant function and would change with time τ ∈ [0 , T ] or location d ∈ D ⊂ R 3 .
Then we should use the SICMDP model with Y =
[0 , T ] or Y = D rather than the ordinary CMDP framework to model such problems:
[Equation]
Load Balancing : Suppose a RL agent needs to balance the load between multiple cell sites using some policy π .
The objective is to minimize the cost V π ( µ ) and the constraint is that at every place d in the region D the cumulative communication capacity C πd ( µ ) is above some threshold u d .
Example 3.2 (Constraints with Uncertainty) .
Again we consider a problem like Problem (1).
In many application scenarios the cost function c ( s, a ) is handcrafted and the construction of c ( s, a ) is not guaranteed to be correct.
Hence it may be helpful to include an additional parameter ϵ ∈ E representing our uncertainty in the construction of the cost function c ( s, a ) as well as the value of constraints u .
Even if the constraint is not handcrafted and has clear physical meaning, it may still subject to uncertain parameters ϵ ∈ E that cannot be observed in advance.
Therefore, we should use the SICMDP model with Y = E rather than the ordinary CMDP framework to model such problems:
[Equation]
Underwater Drone : Suppose an underwater drone needs to maximize V π ( µ ) to accomplish some tasks.
When the unknown environment feature (salinity, temperature, ocean current, etc,) is ϵ ∈ E , for state-action pair ( s, a ) the energy consumption is c
ϵ
( s, a ) , and the constraint is that total energy consumption C πϵ ( µ ) cannot be larger than its battery capacity u ϵ .
Remark 3.3 .
An alternative approach to solving problems such as Examples 3.1 and 3.2 is to naively discretize the constraint set Y , and then the discretized problem can be fit into the conventional
CMDP framework.
The problem of this naive method is that the prior knowledge, i.e., the constraint function is continuous w.r.t.
y , would be lost, which makes the method extremely inefficient.
In
Section 6.2 we demonstrate this issue via a numerical example.
When an SICMDP M is known to us, we may do the planning by solving a linear semi-infinite programming (LSIP) problem.
Denote the occupancy measure on S × A introduced by policy π as q π ∈ ∆( S × A ) .
Then we have
[Equation]
Problem (M) can be reformulated as the following LSIP problem:
[Equation]
Therefore, when M is already known the optimal policy π ∗ can be found by solving Problem (4).
And we always assume such a policy π ∗ exists.
Assumption 3.4.
Problem (M) is feasible with an optimal solution π ∗ , or equivalently, Problem (4) is feasible with an optimal solution q ∗ .
The SI-CRL Algorithm
In this section we present an offline reinforcement learning algorithm called SI-CRL for SICMDPs.
In a high-level point of view, our algorithm is a semi-infinite version of the algorithms proposed in [21, 14].
In the first stage, SI-CRL takes an offline dataset { ( s i , a i , s ′ i ) | i = 1 , 2 , . . .
, m } as input and generate an empirical estimate (cid:98) P of the true transition dynamic P .
Then the algorithm constructs a confidence set (the optimistic set) according to (cid:98) P that would cover the true SICMDP with high probability.
Then for each policy π we would only view its return as the largest possible return in
SICMDPs in the confidence set.
This method is also called the optimistic approach.
In the second stage, the optimistic policy ˜ π is found using a LSIP algorithm.
It can be shown that the resulting policy ˜ π is guaranteed to be nearly optimal, and the theoretical analysis can be found in Section 5.
Now we give a more detailed description of SI-CRL.
First, the empirical estimate (cid:98) P is calculated as: (cid:98) P ( s ′ | s, a ) :
= n ( s,a,s ′ ) max(1 ,n ( s,a )) , where n ( s, a, s ′ ) :
= (cid:80) m i =1 1 { s i
=
s, a i = a, s ′ i =
s ′ } and n ( s, a )
=
(cid:80) s
′ n ( s, a, s ′ ) .
The reason why we do not directly plug (cid:98) P into Problem (4) and solve the resulting
LSIP problem is due to the fact that there is no guarantee that the LSIP problem w.r.t.
(cid:98) P is feasible.
To address this issue, we construct an optimistic set M δ such that with high probability the true
SICMDP M lies in M δ .
In particular, M δ is defined via the empirical Bernstein’s bound and the
Hoeffding’s bound [26]:
[Equation]
where
[Equation]
The next step is to solve the optimistic planning problem:
[Equation]
where the superscript M ′ denotes that the expectation is taken w.r.t.
SICMDP M ′ .
Theorem 4.1.
Suppose n ≥ 3 .
With probability at least 1 − 2 | S | 2 | A | δ , we have that M ∈ M δ , and
Problem (5) is feasible.
The proof is given in the appendix.
Note that the optimization variables include both M ′ and π , and LSIP reformulations like Problem (4) would no longer be possible.
Instead, we shall introduce the state-action-state occupancy measure z
( s, a, s ′ ) .
In particular, assuming z P,π ( s, a, s ′ ) :
=
P ( s ′ | s, a ) q π
( s, a ) , we have P ( s ′ | s, a ) =
[Equation]
and π
[Equation]
a
[Equation]
s
[Equation]
Problem (5) can be reformulated as the following extended LSIP problem:
[Equation]
However, compared to LP problems, LSIP problems are typically harder to solve and there are no all-purpose LSIP solvers.
Here, we choose the simple yet effective dual exchange methods [23, 32] to solve Problem 6.
The SI-CRL algorithm can be summarized in Algorithm 1.
Algorithm 1 SI-CRL
Input: state space S , action space A , dataset { ( s i , a i , s ′ i ) | i = 1 , 2 , ..., m } , reward function r , a continuum of cost function c , value for constraints u , discount factor γ for each ( s, a, s ′ )
tuple do
Set (cid:98) P ( s ′ | s, a ) :
= (cid:80) mi =1 1 {
s i =
s,a i = a,s ′ i = s ′ } max ( 1 , (cid:80) mi
=1 1 { s i =
s,a i = a } ) end for
Initialize Y
= { y 0 } for i = 1 to T do
Use a LP solver to solve a finite version of Problem (6) by only considering constraints in Yand store the solution as z i
Find y i ≈ argmax y ∈ Y (cid:80) s,a,s ′
z i ( s, a, s ′ )
c y
( s, a )
− u y
if (cid:80) s,a,s ′
z ( s, a, s ′ )
c
y i ( s, a ) − u y i ≤ 0 then
Set z T =
z
i
BREAK end if
Add y i to Yend for for each ( s, a ) pair do
Set ˆ π ( a | s ) =
(cid:80) s ′
z T ( s,a,s ′ )
(cid:80) s ′ ,a ′ z T ( s,a ′ ,s ′ )
end for
RETURN ˆ π
Theoretical Analysis
We give PAC-type bounds for SI-CRL under two different settings.
The error of SI-CRL is decomposed into two parts: the statistical error from approximating Problem (M) with Problem (5) and the optimization error from the fact that the solution of (5) obtained by dual exchange method is inexact.
On the statistical side, our goal is to determine that how many samples are required to make
SI-CRL an ( ϵ, δ ) -optimal 1 when Problem (5) can be solved exactly, i.e., we want to find the sample complexity of SI-CRL.
We show that the sample complexity of SI-CRL is (cid:101) O (cid:16) | S | 2 | A | 2 ϵ 2 (1 − γ ) 3 (cid:17) if the dataset we use is generated by a generative model, and (cid:101) O (cid:16) | S ||
A | ν min ϵ 2 (1 − γ ) 3 (cid:17) if the dataset we use is generated by a probability measure ν defined on the space S
× A and P ( ·| s, a ) as considered in [11].
Here (cid:101)
O means that all logarithm terms are discarded, and ν min :=
min ν ( s,a ) >
ν
( s, a ) .
It can be noted that the order of our sample complexity bound increases by a factor of | S
||
A | compared to that of ordinary discounted MDP [8].
On the optimization side, we show that if the inner maximization problem w.r.t.
y can be solved exactly, the dual exchange method would produce an ϵ -optimal solutions 2 when the number of iterations T
= O (cid:18)(cid:104) diam( Y ) L (cid:112) | S | 2 | A | d/ϵ (cid:105) d (cid:19) , where L is the Lipschitz constant defined in Assumption 5.3.
We will present our theoretical analysis in more details in the following part of this section.
5.1 Notation and Preliminaries
Given a stationary policy π , we define the value function V π ( s ) =
E ( (cid:80) ∞ t =0 γ t r ( s t , a t ) | s 0 = s ) ,
V π =
( V π ( s 1 ) , . . .
, V π ( s | S | ))
⊤ ∈ R | S | .
Thus we have V π ( µ ) =
µ ⊤ V π .
Similarly, we define the expected cost C πy
[Equation]
C πy ( µ ) = µ ⊤ C πy .
And π ∗ denotes the optimal policy.
Suppose ˜ π, (cid:102)
M are the solution of Problem ( 5 ) and (cid:102)
M = ⟨ S, A, Y, (cid:101) P, r, c, u, µ, γ ⟩ .
For a given stationary policy π , we use (cid:101) V π ( s ) , (cid:101) V π , (cid:101) V π ( µ ) , (cid:101) C πy ( s ) , (cid:101) C πy , (cid:101) C πy ( µ ) , ˜ q π ( s, a ) , to represent the value function, expected cost, occupancy measure of
SICMDP (cid:102) M , respectively.
We say an offline dataset { ( s i , a
i , s ′ i ) | i = 1 , 2 , . . .
, m } to be generated by a generative model if we sample according to P ( ·| s, a ) for each ( s, a ) -pair n = m/ | S ||
A | times and record the results in the dataset.
We say an offline dataset to be generated by probability measure ν and P ( ·| s, a ) if ( s i , a i ) i.i.d.
∼ ν and s ′ i ∼ P ( ·| s i , a i ) .
An ( ϵ, δ ) -optimal policy is defined as follows.
Definition 5.1.
An RL algorithm is called ( ϵ, δ ) -optimal for ϵ, δ > 0 if with probability at least 1 − δ it can return a policy π such that
[Equation]
An ϵ -optimal solution of Problem (5) is defined as
Definition 5.2.
A stationary policy ˆ π is called an ϵ -optimal solution of Problem (5) for ϵ >
if
[Equation]
hold simultaneously.
Unless otherwise specified, we assume that ∀ ( s, a ) ∈ S × A , c y ( s, a ) is L -Lipschitz in y w.r.t.
∥·∥ 2 .
We also assume that u y is L -Lipschitz in y w.r.t.
∥·∥ 2 .
The assumptions can be formally stated as:
Assumption 5.3.
c y ( s, a ) and u y are Lipschitz in y w.r.t.
∥·∥ 2 , i.e., ∃ L > 0 s.t. ∀
y, y ′ ∈ Y, ( s, a ) ∈
S × A, | c y ( s, a ) − c y ′
( s, a )
| ≤ L ∥ y − y ′
∥ 2 , | u y − u y ′
| ≤ L ∥ y − y ′
∥ 2 .
The Lipschitz assumption is usually necessary when dealing with a semi-infinitely constrained problem [36, 23].
And this assumption is indeed quite mild because Y is a compact set.
5.2 Sample Complexity of SI-CRL
We consider the case where the offline dataset we use is generated by a generative model.
First we consider a restricted setting as in [26] where for each ( s, a ) -pair in the true SICMDP there are at most two possible next-states and provide the sample complexity bound.
Then we will drop Assumption 5.4 using the same strategy as in [26] and derive the sample complexity bound of the general case.
The proof can be found in the appendix.
Assumption 5.4.
The true unknown SICMDP M satisfies P ( s ′ | s, a ) =
for all but two s ′ ∈ S denoted as sa + and sa − ∈ S .
Theorem 5.5.
Suppose Assumption 5.4 holds, and the dataset we use is generated by a generative model with m/ | S
||
A | = n > max (cid:110) 36log4 /
[Equation]
, 4log4 /
[Equation]
3 (cid:111) .
Then with probability 1 − 2 | S | 2 | A | δ , we have that
[Equation]
Theorem 5.6.
Suppose Assumption 5.4 holds, the dataset we use is generated by a generative model and Problem 5 can be solved exactly.
Then when m = O (cid:18) | S ||
A | log ( 8 | S | 2 | A | /δ )
ϵ 2 (1 − γ )(cid:19) , SI-CRL is ( ϵ, δ ) -optimal.
Theorem 5.7.
Suppose the dataset we use is generated by a generative model and Problem 5 can be solved exactly.
Then when m = O (cid:18) | S | 2 | A | 2 (log | S | ) 3 log ( 8 | S | 4 | A | 3 /δ )
ϵ 2 (1 − γ )(cid:19) , a modification of SI-CRL is ( ϵ, δ ) -optimal.
Remark 5.8 .
Our proof strategy is similar to [26].
However, to get a (cid:101) O
((1 − γ ) − 3 ) bound [26] uses a tedious recursion argument.
We greatly simplify the proof and achieve improvements in log terms (by a factor of (log( | S | ϵ (1 − γ ) )) 2 ) using sharper bounds on local variances of MDPs developed in [3].
Remark 5.9 .
Although Assumption 5.4 seems quite restrictive, we argue that it is necessary to establish sharp sample complexity bound, as shown in [26].
Specifically, without this assumption the “quasi-Bernstein bound” (Lemma B.4) will not hold, thus we may not be able to get the (cid:101) O
((1 − γ ) − 3 ) bound.
Remark 5.10 .
It can be noted that our sample complexity bound does not rely on the constraint set Y .
This is because we consider the setting where r and c y are known deterministic functions and the only source of randomness comes from estimating the unknown transition dynamic using an offline dataset.
Now we generalize our results to the case where the offline dataset is generated by a probability measure.
The proof can be found in the appendix.
Theorem 5.11.
Suppose the dataset we use is generated by probability measure
ν
and Problem 5 can be solved exactly.
Then when m =
O (cid:18)
[Equation]
|
[Equation]
min ϵ 2 (1 − γ )(cid:19) , a modification of SI-CRL is ( ϵ, δ ) -optimal.
Remark 5.12 .
Here "a modification of SI-CRL" stands for the following procedure: first we transform the original SICMDP to a new SICMDP satisfying Assumption 5.4, then we run SI-CRL to solve the new SICMDP.
One may refer to the proof in Appendix B for more details.
5.3 Iteration Complexity of SI-CRL
We give the iteration complexity of SI-CRL, i.e., how many iterations are required to output an ϵ -optimal solution of Problem (5) when the inner-loop problem can be solved exactly.
Our results is a corollary of Theorem 4 in [23].
Theorem 5.13.
If the inner-loop maximization problem in SI-CRL can be exactly solved, then
SI-CRL will output an ϵ -optimal solution of Problem (5) if the number of iterations T =
O (cid:18)(cid:110) diam( Y ) L (cid:112) | S | 2 | A | d/ [(1 −
[Equation]
] (cid:111) d (cid:19) .
Numerical Experiments
We design two numerical examples: toy SICMDP and discharge of sewage.
By a set of numerical experiments, we illustrate the SICMDP model and validate the efficacy of the SI-CRL algorithm as well as the correctness of our theoretical results.
We highlight that in the example of discharge of sewage we find that the SICMDP framework greatly outperforms the CMDP baseline obtained by discretizing the original problem in modeling realistic reinforcement learning problems.
We implement our methods with Python and LP problems are solved using a full-featured university version of Gurobi [19].
Details of our implementation can be found in the appendix.
All the experiments are run on a workstation with 8 CPUs and no GPU.
6.1 Toy SICMDP
We consider a most simple SICMDP with | S | = 2 , | A | = 2 and Y =
Its MDP part is specified in Figure 1, where p ∈ (0 . 5 , 1) and τ ≪ 1 is a small positive number.
For each γ ∈ (0 , 1) , we design
Lipschitz c y and u y such that the optimal policy takes a 0 with probability 0.9 and 0.5 on s 0 and s 1 , respectively.
For details of the construction of Toy SICMDP, one may refer to the appendix.
To make our numerical results more reliable, we repeat all experiments in this subsection for 30 times and report the average results.
First, we would like to check the efficacy of the SI-CRL algorithm.
We set
T sufficiently large such that the algorithm is guaranteed to converge.
Then we gradually increase m , the size of the dataset, and see whether SI-CRL can recover the pre-defined optimal policy.
The results are shown in Figure 3.
It can be noticed that as m gets larger, the error term converges to zero, showing that our SI-CRL algorithm may effectively solve reinforcement learning problems for
SICMDPs.
Second, we would like to validate the theoretical results in Section 5.
Specifically, we investigate the sample complexity of SI-CRL for a fixed ( ϵ, δ ) (See Definition 5.1) when γ and ν min vary.
T is set to be sufficiently large as in the previous experiment.
We present the results in Figure 3.
The logarithm of sample complexity vs. the transformed parameter of interest is approximately linear with slope 1, which indicates our sample complexity bounds are correct and tight.
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
6.2 Discharge of Sewage
To demonstrate the power of the SICMDP model and the SI-CRL algorithm we consider a more realistic and complex problem adapted from [18].
Assume there are | S | sewage outfalls in a region [0 , 1] d , with d = 2 or 3 , and at each time point only one single outfall is active.
The active outfall would cause pollution in nearby areas, and the impact would decrease with Euclidean distance.
We need to figure out a strategy to switch between neighboring outfalls to avoid over-pollution at each location of the region while minimizing the switching cost.
Clearly, this problem can be formulated as a SICMDP model with Y =
[0 , 1] d and corresponding c y and u y .
For details of the construction of the Discharge of Sewage, one may refer to the appendix.
In the following numerical experiments, we assume that an offline dataset generated by a generative model is available.
First, we numerically validate our theoretical bounds on sample complexity and iteration complexity.
In particular, we investigate the relationship between the sample complexity and iteration complexity of SI-CRL and the size of state space | S | .
Like the case in Toy SICMDP, we find the numerical results fit well with our theoretical analysis.
We show the results in Figures 4.
As before, we run each experiment for 30 times and report the averaged results.
Second, we compare our method with a naive CMDP baseline 3.3, showing the advantage of SICMDP in modeling problems like Example 3.1, 3.2.
In the baseline method, we only consider the constraints on a grid of Y containing T baseline points, which allows us to model Discharge of Sewage as a standard
CMDP problem with T baseline constraints.
The CMDP problem is then solved by the algorithm proposed in [14].
We visualize the quality of solutions of our proposed method and baseline method in Figure 5.
It can be found that when T = T baseline , the policy obtained by our proposed methods is of far better quality than the policy obtained by the baseline methods.
[Figure]
l o g
[Figure]
|
[Figure]
l o g
[Figure] [Figure]
[Figure]
An anti-intuitive phenomenon is that although in our method we need to deal with multiple LP problems while in the baseline we only solve one single LP problem, our method is still more time-efficient than the CMDP baseline.
Figure 7 indicates that our method takes less time to get a solution of given accuracy, which is evaluated by the error term sup y ∈ Y C
ˆ πy ( µ )
− u y .
The reason is that in SI-CRL we solve LP problems with a dual simplex method, thus re-optimization after adding a new constraint is much faster than re-solving the LP problem from scratch[25].
And our method needs far fewer active constraints to attain the same accuracy as the baseline methods, see Figure 6.
Conclusion
We have studied a novel generalization of CMDP that we have called SICMDP.
In particular, we have considered a continuum of constraints rather than a finite number of constraints.
We have devised a reinforcement learning algorithm SI-CRL to solve SICMDP problems.
Furthermore, we have presented theoretical analysis for SI-CRL, establishing the sample complexity bounds as well as the iteration complexity bounds.
We have also performed the extensive numerical experiments to show the efficacy of our proposed method and its advantage over traditional CMDPs.
However, the SI-CRL algorithm can only handle the tabular case, with a nice offline dataset avaliable.
We would study the
SICMDP beyond the tabular case and develop efficient algorithms in future works.
Acknowledgments and Disclosure of Funding
This work has been supported by the National Key Research and Development Project of China (No.
2020AAA0104400).
Also, the authors would like to thank Mr. Hao Jin for helpful discussions.
B Omitted Proofs in Section
First we define some additional notations.
We use Q π ( s, a ) :
=
E ( (cid:80) ∞ t =0 γ t r ( s t , a )) to denote the state-action value function.
The local variance is defined as Var P ( V π )( s, a ) =
E s ′ ∼ P ( ·| s,a ) ( V π ( s ′ ) −
P ( ·| s, a ) V π ) 2 .
We view V π as vector of length | S | and Q π , r, Var P
( V π ) as vectors of length | S |·| A | .
We overload notation and let P also refer to a matrix of size ( | S |·| A | ) ×| S | , where the entry P ( s,a ) ,s ′ is equal to P ( s ′ | s, a ) .
We also define P π to be the transition matrix on state-action pairs induced by a stationary policy π , namely:
[Equation]
We use (cid:101) Q π ( s, a ) , Var (cid:101) P ( (cid:101) V π )( s, a ) , (cid:101) Q π , Var (cid:101) P ( (cid:101) V π ) , (cid:101) V π , (cid:101) σ 2 π , (cid:101) P, (cid:101) P π to denote the state-action value function, local variance, vector of state-action value function, vector of local variance, vector of value function, transition matrix, transition matrix on state-action pairs w.r.t.
SICMDP (cid:102) M , respectively.
Lemma B.1.
If Assumption 5.4 is true and M ∈ M δ , we have
[Equation]
Proof.
Given a stationary policy π , if Assumption 5.4 is true and M ∈ M δ ,
[Equation]
which implies
[Equation]
/δ
[Equation]
Then we have
[Equation]
Lemma B.2.
Given a stationary policy π , when Assumption 5.4 is true and M ∈ M δ , we have
[Equation]
Proof.
For simplicity of notation, we drop the dependence on π .
By definition,
[Equation]
where ( · ) 2 means element-wise squares.
When Assumption 5.4 is true and M ∈ M δ , by Lemma B.1,
[Equation]
≤
[Equation]
δ 2 n .
We also have
[Equation]
inequality)
[Equation]
P ( (cid:101)
[Equation] [Equation]
B.1) .
Therefore, we can get
Var P
[Equation]
V π )
[Equation]
γ
[Equation]
/δ
[Equation]
n + 8 γ
[Equation]
(1 − γ )log 2 /δ 2 n .
Lemma B.3.
Let p, ˜ p, ˆ p ∈ [0 , 1] satisfy | p − ˆ p | ≤ min (cid:40)(cid:114) 2ˆ p (1 − ˆ p ) log 4 /δ n + 4 log 4 /δ n , (cid:114) log 2 /δ 2 n (cid:41) | ˜ p − ˆ p | ≤ min (cid:40)(cid:114) 2ˆ p (1 − ˆ p ) log 4 /δ n + 4 log 4 /δ n , (cid:114) log 2 /δ 2 n (cid:41) .
Then | p − ˜ p | ≤ (cid:114) 8 p (1 − p ) log 4 /δ n +(cid:18) log 4 /δ
[Equation]
n
Proof.
Assume WLOG that ˆ p ≥ p .
Therefore,
[Equation]
Similarly, we have
[Equation]
Thus we may complete the proof using triangular inequality.
Lemma B.4.
Given a stationary policy π , suppose Assumption 5.4 is true and M ∈ M δ , then
[Equation]
where ⪯ means every element of LHS is less than or equal to the its counterpart in RHS.
Proof.
Let p =
P ( sa + | s, a ) , ˜ p
= ˜ P
( sa + | s, a ) .
Applying Lemma B.3 yields
[Equation]
Assume WLOG that V π ( sa + ) ≥ V π ( sa − ) .
Therefore we have
[Equation]
Since p (1 − p
[Equation]
( V π ( sa + ) − V π ( sa − )) 2 =
[ pV π ( sa + ) 2 + (1 − p )
V π ( sa − ) 2 ] − [ pV π ( sa + ) + (1 − p )
V π ( sa − )]=
Var P ( V π )
We may get |
[Equation]
P
[Equation]
a
[Equation]
n
[Equation]
1 − γ
[Equation]
, which completes the proof.
Lemma B.5.
Given a stationary policy π , suppose Assumption 5.4 is true and M ∈ M δ , then we have
[Equation]
Proof.
From Lemma C.3, Lemma C.4, Lemma C.5, Lemma B.4 and the fact that (cid:16) I − γ (cid:101) P π (cid:17) − 1 has positive entries, we know
[Equation]
1 (cid:112) Var P
V π ) (cid:13)(cid:13)(cid:13)(cid:13) ∞
+ (1 − γ )(cid:18) log 4 /δ n (cid:19) 3 /+
(1 − γ )(cid:18) log 4 /δ n (cid:19) ≤ (cid:114) 16 log 4 /δ n (cid:13)(cid:13)(cid:13) (cid:13) (cid:16) I − γ (cid:101) P π (cid:17) − 1 (cid:113)
Var (cid:101) P ( (cid:101)
V π )
(cid:13)(cid:13)(cid:13) (cid:13) ∞ + 4 √(1 − γ )(cid:18) log 4 /δ n (cid:19) 3 /+
(1 − γ )(cid:18) log 4 /δ n (cid:19) ≤ (1 − γ ) 3 /(cid:114) log 4 /δ n + 4 √(1 − γ )
(cid:18) log 4 /δ n (cid:19) 3 /+
(1 − γ )(cid:18) log 4 /δ n (cid:19) .
The proof is completed since (cid:13)(cid:13)(cid:13) V π − (cid:101)
V π (cid:13)(cid:13)(cid:13) ∞ ≤ (cid:13)(cid:13)(cid:13) Q π − (cid:101) Q
π (cid:13)(cid:13)(cid:13) ∞ by definitions.
Lemma B.6.
Suppose Assumption 5.4 is true and n > 6log4 /δ
(1 − γ ) 5 / 2 , then with probability at least 1 − 2 | S | 2 | A | δ , we have
[Equation]
Proof.
When Assumption 5.4 is true and M ∈ M δ , it follows from Lemma B.5 that
[Equation]
And by setting n > max (cid:110) 36log4 /δ
(1 − γ ) 2 , 4log4 /δ
(1 − γ ) 3 (cid:111)
we will get (cid:13)(cid:13) (cid:13) V π −
[Equation]
(cid:13) ∞ ≤
[Equation]
log
[Equation]
n
[Equation]
1 − γ ) 3 .
Similar arguments can be applied to bound (cid:13)(cid:13)(cid:13) C πy − (cid:101) C πy (cid:13)(cid:13)(cid:13) ∞ .
Since by Theorem 4.1 we have
[Equation]
≥ 1 − 2 | S | 2 | A | δ, the proof is completed.
Proof of Theorem 5.5.
By Lemma B.5, we know that with probability 1 − 2 | S | 2 | A | δ ,
[Equation]
[Equation] [Equation]
n
[Equation]
Thus | V
[Equation]
[Equation] [Equation]
n
[Equation] [Equation]
Noting that ˜ V ˜ π ( µ ) ≥ ˜
V
π ∗ ( µ ) , we may get
[Equation] [Equation]
Similarly, when
[Equation]
, we may get
[Equation]
since ˜ C ˜ πy ( µ ) ≤ u y .
Proof of Theorem 5.6.
Theorem 5.6 is a direct corollary of Theorem 5.5.
Proof of Theorem 5.7.
The proof is nearly identical to the proof of Theorem 3 in [26].
The idea is to augment each state/action pair of the original mdp with | S | − 2 states in the form of a binary tree as pictured in the diagram below.
The intention of the tree is to construct an SICMDP, ¯ M = ⟨ ¯ S, A, Y, ¯ P, ¯ r, ¯ c, u, µ, ¯ γ ⟩ that with appropriate transition probabilities is functionally equivalent to M while satisfying Assumption 5.4.
The rewards and costs in the added states are set to zero.
Since the tree has depth d = O (log 2 | S | ) , it now takes d time-steps in the augmented SICMDP to change states once in the original SICMDP.
Therefore we must also rescale the discount factor ¯ γ by setting ¯ γ < γ d .
Now we have
[Equation]
Then we complete the proof by applying results in Theorem 5.6.
Proof of Theorem 5.11.
By Theorem C.6, we have for any fixed ( s, a ) ∈ S ×
A
[Equation] [Equation]
−
[Equation]
m
Let m =
[Equation]
, we have P ( n ( s, a ) ≥ mν min )) ≥ 1 − δ/
| S ||
A | .
Therefore, with probability at least 1 − δ/ 2 , we can get
[Equation]
Then our problem is reduced to the case that the offline dataset is generated by generative models.
The proof is completed by using results in Theorem 5.7.
Proof of Theorem 5.13.
See Theorem 4 in [23].
C Auxiliary Lemmas
Lemma C.1 (Empirical Bernstein Inequality) .
Suppose n ≥ 3 , { X 1 , ..., X n } be n i.i.d.
random variables with values in [0 , 1] .
Let δ > 0 .
Then with probability at least 1 − δ we have
[Equation]
where V n ( X 1:
[Equation]
) :=
[Equation]
denotes the empirical variance of the dataset { X 1 , ..., X n } .
Proof.
See Theorem 11 in [28].
Lemma C.2 (Hoeffding’s Inequality) .
Suppose { X 1 , ..., X n } be n i.i.d.
random variables with values in [0 , 1] .
Let δ > 0 .
Then with probability at least 1 − δ we have
[Equation]
Proof.
See Theorem 2.2.6 in [38].
Lemma C.3.
For any policy π and transition probabilities P , (cid:101) P , we have that
[Equation]
Proof.
See Lemma 2 in [3].
Lemma C.4.
For any policy π , any transition probability P and any vector v ∈ R | S |·| A | , we have
[Equation]
Proof.
See Lemma 3 in [3].
Lemma C.5.
For any policy π and any transition probability P , we have
[Equation]
where √· is defined as the element-wise square root.
Proof.
See Lemma 4 in [3].
Theorem C.6 (Chernoff’s Inequality) .
Let X i be independent Bernoulli random variables with parameter p i .
Consider their sum S N = (cid:80) Ni
=1 X i and denote its mean by µ = E S N .
Then, for any t < µ , we have
[Equation]
Proof.
See [38].
D Details of Numerical Experiments
We will release codes of our algorithms and construction of corresponding environments on GitHub soon.
D.1 Construction of Toy SICMDP
Without loss of generality, assume Y =
We split Y =
[0 , 3] to Y 1 =
[0 , 1] , Y 2 =
[1 , 2] and
Y 3 =
Intuitively, in Y 1 ( Y 3 ), we restrict the agent to take action a 0 in s 0 ( s 1 ) with the given probability from the target policy ˜ π .
(Recall, taking action a 0 is always better if we set aside the constraints.)
We introduce Y 2 to obtain L -Lipschitz of c y and u y .
Assume f : [ − 0 .
→ R is a continuous-differentiable even function, with unique maximum point 0 and zero point 0 .
5 , we use f ( x ) =
(1 + cos(2 πx )) cos(2 πx ) in practice.
Let c y ( s 0 , a 0 ) =
f ( y − 0 . 5) for y ∈ Y 1 ,
c y ( s 1 , a 0 ) =
f ( y − 2 . 5) for y ∈ Y 3 , and c ≡ 0 otherwise.
For each γ ∈ (0 , 1) , we define u 1 = C ˜ π 0 .
5 > 0 and u 1 = C ˜
π 2 .
5 > 0 .
Let u y ≡ u 1 in Y 1 , u y ≡ u 2 in Y 3 and we make linear interpolate u y in Y 2 .
So far, we have constructed Lipschitz c y and u y .
The only active constraints are C π 0 .
5 ≤ u 1 and
C π 2 .
5 ≤ u 2 and the optimal policy is the known target policy ˜ π .
D.2 Construction of Discharge of Sewage
In the numerical experiments, we generate this environment randomly.
In each experiment, we sample the environments 30 times and report the average result.
Positions of sewage outfalls, transition dynamics and rewards are sampled uniformly on Y , probability simplex and [0 , 1] respectively.
Assume f : [0 , + ∞ ) → [0 , + ∞ ) is a continuous-differentiable decreasing function, we use f ( x ) =1+ x 2 in practice.
Let c y
( s, a ) =
c y ( s ) =
f ( ∥ y − s ∥ 2 ) , where s also represents the position of the state (outfall).
Given a target state-occupancy measure d which can be generated randomly or specified in advance, we define u y =
(1 + ∆) (cid:80) s
∈ S d ( s ) c y ( s ) , where ∆ is a small positive number.
The feasibility of the resulting SICMDP is not guaranteed even if ∆ = 0 , we reject those infeasible environments and re-sample.
The SICMDP would be nontrivial if we choose a suitable ∆ .
D.3 Implementation of SICMDP
The only thing we need to clarify is how to solve the maximization problem to generate new y i in i th iteration.
Since the cost function c y ( s, a ) can be non-convex in y , we choose to solve the maximization problem by brute force.
Specifically, we first create a grid of Y of size 10 5 , and then find the grid point with max objective.
This method works well since in the problems we consider Y is of low dimensions.