D ENSITY C ONSTRAINED R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Constrained reinforcement learning (CRL) plays an important role in solving safety-critical and resource-limited tasks.
However, existing methods typically rely on tuning reward or cost parameters to encode the constraints, which can be tedious and tend to not generalize well.
Instead of building sophisticated cost functions for constraints, we present a pioneering study of imposing constraints directly on the state density function of the system.
Density functions have clear physical meanings and can express a variety of constraints in a straightforward fashion.
We prove the duality between the density function and Q function in
CRL and use it to develop an effective primal-dual algorithm to solve density constrained reinforcement learning problems.
We provide theoretical guarantees of the optimality of our approach and use a comprehensive set of case studies including standard benchmarks to show that our method outperforms other leading CRL methods in terms of achieving higher reward while respecting the constraints.
I NTRODUCTION
Constrained reinforcement learning (CRL) (Achiam et al., 2017; Altman, 1999; Dalal et al., 2018;
Paternain et al., 2019; Tessler et al., 2019) has received increasing interests as a way of addressing the safety challenges in reinforcement learning (RL).
CRL techniques aim to ﬁnd the optimal policy that maximizes the cumulative reward signal while respecting the speciﬁed constraints.
Existing CRL approaches typically involve constructing suitable cost functions and value functions to take into account the constraints.
Then a crucial step is to choose appropriate parameters such as thresholds for the cost and value functions to encode the constraints.
However, one signiﬁcant gap between the use of such methods and solving practical RL problems is the correct construction of the cost and value functions, which is typically not solved systematically but relies on engineering intuitions (Paternain et al., 2019).
Simple cost functions may not exhibit satisfactory performance, while sophisticated cost functions may not have clear physical meanings.
When cost functions lack clear physical interpretations, it is difﬁcult to formally guarantee the satisfaction of the performance speciﬁcations, even if the constraints on the cost functions are fulﬁlled.
Moreover, different environments generally need different cost functions, which makes the tedious tuning process extremely time-consuming.
In this work, we ﬁll the gap by imposing constraints on the state density functions as an intuitive and systematic way to encode constraints in RL.
Density is a measurement of state concentration in the state space, and is directly related to the state distribution.
It has been well-studied in physics (Yang, 1991) and control (Brockett, 2012; Chen & Ames, 2019; Rantzer, 2001).
A variety of real-world constraints are naturally expressed as density constraints in the state space.
Pure safety constraints can be trivially encoded as the entire density of the states being contained in the safe region.
In more general examples, the vehicle densities in certain areas are supposed to be less than the critical density (Gerwinski & Krug, 1999) to avoid congestion.
When spraying pesticide using drones, different parts of a farmland requires different levels of pesticide density.
Indeed, in the experiments we will see how these problems are solved with guarantees using density constrained RL (DCRL).
Our approach is based on the new theoretical results of the duality relationship between the density function and value function in optimal control (Chen & Ames, 2019).
One can prove generic duality between density functions and value functions for both continuous dynamics and discrete-state
Markov decision processes (MDP), under various setups such as using Bolza form terminal constraints, inﬁnite horizon discounted rewards, or ﬁnite horizon cumulative rewards.
In Chen & Ames (2019) the duality is proved for value functions in optimal control, assuming that the full dynamics of the world model is known.
In this paper, we take a nontrivial step to establish the duality between the density function and the Q function (Theorem 1).
We also reveal that under density constraints, the density function and Q function is also dual to each other (Theorem 2), which enables us to enforce constraints on state density functions in CRL.
We propose a model-free primal-dual algorithm (Algorithm 1) to solve the DCRL problem, which is applicable in both discrete and continuous state and action spaces, and can be ﬂexibly combined with off-the-shelf RL methods to update the policy.
We prove the optimality of the policies returned by our algorithm if it converges (Proposition 1).
We also discuss the approaches to computing the key quantities required by Algorithm 1.
Our main contributions are: 1) We are the ﬁrst to introduce the DCRL problem with constraints on state density, which is associated with a clear physical interpretation.
We are the ﬁrst to prove and use the duality between density functions and Q functions over continuous state space to solve
DCRL.
Our model-free primal-dual algorithm solves DCRL and can guarantee the optimality of the reward and satisfaction of density constraints simultaneously.
We use an extensive set of experiments to show the effectiveness and generalization capabilities of our algorithm over leading approaches such as CPO, RCPO, and PCPO, even when dealing with conﬂicting requirements.
Related work.
Safe reinforcement learning (Garcıa & Fern´andez, 2015) primarily focuses on two approaches: modifying the optimality criteria by combining a risk factor (Heger, 1994; Nilim &
El Ghaoui, 2005; Howard & Matheson, 1972; Borkar, 2002;
Basu et al., 2008; Sato et al., 2001;
Dotan Di Castro & Mannor, 2012; Kadota et al., 2006; L ¨ otjens et al., 2019) and incorporating extra knowledge to the exploration process (Moldovan & Abbeel, 2012; Abbeel et al., 2010; Tang et al., 2010; Geramifard et al., 2013; Clouse & Utgoff, 1992; Thomaz et al., 2006;
Chow et al., 2018).
Our method falls into the ﬁrst category by imposing constraints and is closely related to constrained
Markov decision processes (Altman, 1999) (CMDPs) and CRL (Achiam et al., 2017; Lillicrap et al., 2016).
CMDPs and CRL have been extensively studied in robotics (Gu et al., 2017; Pham et al., 2018), game theory (Altman & Shwartz, 2000), and communication and networks (Hou & Zhao, 2017; Bovopoulos & Lazar, 1992).
Most previous works consider the constraints on value functions, cost functions and reward functions (Altman, 1999; Paternain et al., 2019; Altman & Shwartz, 2000;
Dalal et al., 2018; Achiam et al., 2017; Ding et al., 2020).
Instead, we directly impose constraints on the state density function.
Our approach builds on Chen et al.
and Chen & Ames (2019), which assume known model dynamics.
Instead, in this paper we consider the model-free setting and proved the duality of density functions to Q functions.
In Geibel & Wysotzki (2005) density was studied as the probability of entering error states and thus has fundamentally different physical interpretations from us.
In Dai et al.
(2017) the duality was used to boost the actor-critic algorithm.
The duality is also used in the policy evaluation community (Nachum et al., 2019; Nachum & Dai, 2020; Tang et al., 2019).
The ofﬂine policy evaluation method proposed by Nachum et al.
(2019) can also be used to estimate the state density in our paper, but their focus is policy evaluation rather than constrained RL.
Therefore, we claim that this paper is the ﬁrst work to consider density constraints and use the duality property to solve CRL.
P RELIMINARIES
Markov Decision Processes (MDP).
An MDP M is a tuple (cid:104) S, A, P, R, γ (cid:105) , where (1) S is the (possibly inﬁnite) set of states; (2) A is the (possibly inﬁnite) set of actions; (3) P : S ×
A × S (cid:55)→ [0 , 1] is the transition probability with P ( s, a, s (cid:48) )
the probability of transitioning from state s to s (cid:48) when action a ∈ A is taken; (4) R : S ×
A × S (cid:55)→ R is the reward associated with the transition P under the action a ∈ A ; (5) γ ∈ [0 , 1] is a discount factor.
A policy π maps states to a probability distribution over actions where π ( a | s ) denotes the probability of choosing action a at state s .
Let a function φ : S (cid:55)→ R speciﬁes the initial state distribution.
The objective of an MDP optimization is to ﬁnd the optimal policy that maximizes the overall discounted reward J p
= (cid:82) S
φ ( s ) V π ( s ) ds , where V π ( s ) is called the value function and satisﬁes V π ( s ) =
r π ( s ) + γ (cid:82)
A π ( a | s ) (cid:82) S P ( s, a, s (cid:48) )
V π ( s (cid:48) ) ds (cid:48) da , and r π ( s ) =
(cid:82)
A π ( a | s ) (cid:82) S P ( s, a, s (cid:48) )
R ( s, a, s (cid:48) )
ds (cid:48)
da is the one-step reward from state s following policy π .
For every state s with occurring as an initial state with probability φ ( s ) , it incurs a expected cumulative discounted reward of V π ( s ) .
Therefore the overall reward is (cid:82) S φ ( s ) V π ( s ) ds .
Although the equations are written in integral forms corresponding to continuous state-action space, the discrete counterparts can be derived similarly.
Two major methods for solving MDPs are value iteration and policy iteration, both based on the Bellman operator.
Take value iteration as example, the Bellman optimality condition indicates V ( s )
=
max a ∈ A (cid:82) S P ( s, a, s (cid:48) )
( R ( s, a, s (cid:48) )
+ γV ( s (cid:48) ))
ds (cid:48) .
However, the formulation of value functions in MDP typically cannot handle constraints on the state distribution, which motivates the density functions.
Density Functions.
Stationary state density functions ρ s : S (cid:55)→ R ≥ 0 are measurements of the state concentration in the state space (Chen & Ames, 2019; Rantzer, 2001).
1 We will show later that generic duality relationship exists between density functions and value functions (or Q functions), which allows us to directly impose density constraints in RL problems.
For inﬁnite horizon MDPs, a given policy π and an initial state distribution φ , the stationary density of state s is expressed as:
[Equation]
which is the discounted sum of the probability of visiting s .
The key motivation behind the stationary density distribution is because most of the density constraints are time invariant and are instead deﬁned over the state space.
Stationary distribution gives a projection of all possible reachable states to the state space and is invariant over time.
Therefore, as long as the stationary density satisfy the density constraints, the system always satisﬁes the density constraints at any time.
It is straightforward to show that all theories and algorithms in this paper actually does not depend on the fact that the density distribution is stationary.
We prove in the appendix that the density has an equivalent expression
[Equation]
where φ is the initial state distribution.
φ coincides with the normalized (positive) supply function in Chen et al.
(2019), which is deﬁned as the rate of state
s entering the state space as an initial condition.
We omit the details on constructing stationary distribution, which are provided in Chen et al.
Deﬁnition 1 (DCRL) .
Given an MDP M = (cid:104) S, A, P, R, γ (cid:105) and an initial state distribution φ , the density constrained reinforcement learning problem ﬁnds the optimal policy π (cid:63) that maximizes the expectation of the cumulative sum of a reward signal (cid:82) S φ ( s ) V π ( s ) ds , subject to constraints on the stationary state density function represented as ρ min ( s ) ≤ ρ π (cid:63) s ( s ) ≤ ρ max ( s ) for ∀ s ∈ S .
D ENSITY CONSTRAINED REINFORCEMENT LEARNING
In this section, we ﬁrst show the duality between Q functions and density functions, and then introduce a primal-dual algorithm for solving DCRL problems.
3. D UALITY BETWEEN DENSITY FUNCTIONS AND Q FUNCTIONS
The duality relationship between density functions and value functions in optimal control is recently studied, for both dynamical systems and ﬁnite-state MDPs in Chen & Ames (2019).
In this paper, we take a step further to show the duality property between the density function and Q function, for both continuous and discrete state MDPs.
Our work is the ﬁrst to prove and use the duality between density functions and Q functions over continuous state space to solve density constrained RL, which has not been explored and utilized by published RL literature.
We use the standard inﬁnite horizon discounted rewards case as an illustrative example.
The duality in other setups such as ﬁnite horizon cumulative rewards and inﬁnite horizon average rewards can be proved in a similar way.
To show the duality with respect to Q function, we extend the stationary density ρ πs to consider the action taken at each state.
Let ¯ ρ s : S ×
A → R ≥ 0 be a stationary state-action density function, which represents the amount of state instances taking action a at state s .
¯ ρ s is related to ρ s via marginalization: ρ s ( s ) = (cid:82) A ¯
ρ s ( s, a ) da .
Under a policy π , also have ¯ ρ πs ( s, a )
=
ρ πs ( s ) π ( a | s )
Let r ( s, a ) =
(cid:82) S P ( s, a, s (cid:48) )
R ( s, a, s
[Equation]
) ds
[Equation]
Consider the density function optimization:
[Equation]
and the Q function optimization:
[Equation]
Theorem 1.
The optimization objectives J d and J p are dual to each other and there is no duality gap.
If both are feasible, then J d = J p
and they share the same optimal policy π (cid:63) .
The proof of Theorem 1 is included in the appendix.
Theorem 1 states that optimizing the density function (dual) equals to optimizing Q functions (primal) since they will result in the same optimal policy π (cid:63) .
Since the dual optimization is directly related to density, we can naturally enforce density constraints on the dual problem.
Consider the density constrained optimization:
[Equation]
The marginalization ρ πs ( s ) =
(cid:82)
A ¯ ρ πs ( s, a ) da .
Denote the Lagrange multipliers for ρ πs ≥ ρ min and ρ πs ≤ ρ max as σ − : S (cid:55)→ R ≥ 0 and σ + : S (cid:55)→ R ≥ 0 .
The primal problem is formulated as:
[Equation]
The difference between (6) and (4) is that the reward r ( s, a ) is adjusted by σ − ( s ) − σ + ( s ) .
Theorem 2.
The density constrained optimization objectives J cd and J cp are dual to each other.
If both are feasible and the KKT conditions are satisﬁed, then they share the same optimal policy π (cid:63) .
The proof of Theorem 2 is provided in the appendix.
Theorem 2 reveals that when the KKT conditions are satisﬁed, the optimal solution to the adjusted unconstrained primal problem (6) is exactly the optimal solution to the dual problem (5) with density constraints.
Such an optimal solution not only satisﬁes the state density constraints, but also maximizes the the total reward J cd .
Thus it is the optimal solution to the DCRL problem.
Remark 1.
The duality relationship between density and value function wide exists, for example, in Linear Programming.
Technically, any nonnegative dual variable satisfying the conservation law (Liouville PDE in the continuous case, see Chen & Ames (2019)) is a valid dual.
However, among all valid dual variables, the state density is associated with a clear physical interpretation as the concentration of states, and we are able to directly apply constraints on the density in RL.
Remark 2.
Theorem 1 and 2 can be used in Constrained MDP (CMDP).
In CMDP, the constraint J c = (cid:82) S µ ( s ) V C ( s ) ds ≤ α is indeed a special case of the density constraint.
J c = (cid:82)
S µ
[Equation]
s
[Equation]
s
[Equation]
s ) r c ( s ) ds ≤ α , where r c is the immediate cost and V c is the value of cost.
Thus the CMDP constraint J c ≤ α equals to a special case of constraint on density ρ s .
−
[Figure]
3. P RIMAL - DUAL ALGORITHM FOR DCRL
The DCRL problem looks for an optimal policy of MDP subject to constraints on the density function.
In this section, we continue considering the case where the stationary density in the inﬁnite horizon discounted reward case must satisfy some upper and lower bounds: ρ max ( s ) ≥ ρ s ( s ) ≥ ρ min ( s ) , ∀ s ∈ S .
However, the primal-dual algorithm discussed in this section can be easily extended to handle other reward function setup and other types of constraints on the density function.
By utilizing the duality between density function and Q function (Theorem 2) in the density constrained optimization (5-6), the DCRL problem can be solved by alternating between the primal and dual problems, as is illustrated in Figure 1.
In the primal domain, we solve the adjusted primal problem (reward adjusted by Lagrange multipliers) in (6) using off-the-shelf unconstrained RL methods such as TRPO (Schulman et al., 2015) and DDPG (Lillicrap et al., 2016).
Note that the density constraints are enforced in dual domain and the primal domain is still an unconstrained problem, which means we can make use of existing RL methods to solve the primal problem.
In the dual domain, the policy is used to evaluate the state density function, which is described in details in Section 3.3.
If the KKT conditions σ + · ( ρ πs − ρ max ) = 0 , σ − · ( ρ min − ρ πs )
=
and ρ min ≤ ρ πs ≤ ρ max are not satisﬁed, the Lagrange multipliers are updated and enter the next primal-dual loop.
The key insight is that the density constraints can be enforced in the dual problem, and we can solve the dual problem by solving the equivalent primal problem using existing algorithms.
Alternating between the primal and dual optimization can gradually adjust the Lagrange multipliers until the KKT conditions are satisﬁed.
A general template of the primal-dual optimization with density constraint is provided in Algorithm 1.
Algorithm
Update
Generate experience D
∼ φ a ∼ π
Compute stationary
10: σ − ← max( σ − + α (
− ρ πs ))
12: Return
In Lines 5-6 of Algorithm 1, the Lagrange multipliers σ + and σ − are used to adjust rewards, which lead to an update of the policy π .
In Lines 7-8, the policy is used to evaluate the stationary density, then the Lagrange multipliers are updated following dual ascent (Lines 9-10).
The iteration stops when all the KKT conditions are satisﬁed.
Although Algorithm 1 is derived for the inﬁnite horizon reward case, it also applies to the ﬁnite horizon case.
Proposition 1.
If Algorithm 1 converges to a feasible solution that satisﬁes the KKT condition, it is the optimal solution to the DCRL problem.
(Proof provided in the appendix.)
3. C OMPUTATIONAL APPROACHES
Algorithm 1 requires computing the policy π , stationary density ρ πs , Lagrange multipliers σ + and σ − .
For π , there are well-developed representations such as neural networks and tabular methods.
Updating π using experience D π is also straightforward using standard approaches such as policy gradients or Q-Learning.
By contrast, the computation of ρ πs , σ + and σ − is need to be addressed.
The following computational approaches apply to both ﬁnite and inﬁnite horizon.
Density functions.
In the discrete state case, ρ πs is represented by a vector where each element corresponds to a state.
To compute ρ πs from experience D π (line 8 of Algorithm 1), let D π contain
N episodes where episode i ends at time T i .
Let s ij represent the state reached at time j in the i th episode.
Initialize ρ πs ← 0 .
For all i ∈ { 1 , · · · , N } and j ∈ { 0 , 1 , · · · , T i } , do the update ρ πs ( s ij ) ← ρ πs ( s ij ) + 1 N γ j .
The resulting vector ρ πs approximates the stationary state density.
In the continuous state space, ρ πs cannot be represented as a vector since there are inﬁnitely many states.
We utilize the kernel density estimation method Chen (2017); Chen & Ames (2019) that computes ρ πs ( s ) at state s using the samples in D π with ρ πs ( s )
=
1 N (cid:80) Ni =1 (cid:80) T i j
=0 γ j K h ( s − s ij ) where K h is the kernel function satisfying ∀ s ∈ S, K h ( s ) ≥ 0 and (cid:82) S K h ( s ) ds = 1 .
There are multiple choices of the kernel K h , e.g. Gaussian, Spheric, and Epanechnikov kernels Chen (2017), and probabilistic guarantee of accuracy can be derived, c.f. Wasserman (2019).
Lagrange multipliers.
If the state space is discrete, both σ + and σ − are vectors whose length equals to the number of states.
In each loop of Algorithm 1, after the stationary density is computed, σ + and σ − are updated following Line 9 and Line 10 respectively in Algorithm 1.
If the state space is continuous, we construct Lagrange multiplier functions σ + and σ − from samples in the state space leveraging linear interpolation.
Let ¯ s =
[ s 1 , s 2 , · · · ] represent the samples in the state space.
In every loop of Algorithm 1, denote the Lagrange function computed by the previous loop as σ o + and σ o − .
We compute the updated Lagrange multipliers at states ¯ s as:
[Equation]
Then the new σ + and σ − are obtained by linearly interpolating ¯ σ + and ¯ σ − respectively.
E XPERIMENT
We consider the MuJoCo (Todorov et al., 2012) benchmark and the autonomous electrical vehicle routing benchmark adopted from Blahoudek et al.
(2020) in our experiment.
We also report additional experimental results on 3 other benchmarks in the appendix, including a safe electrical motor control, an agricultural spraying drone, and a express delivery service transportation system, which all can show the power of our DCRL method over other approaches when dealing with complex density constraints.
The evaluation criteria includes the reward and constraint values .
The methods should keep the constraint values below the required thresholds and achieve as much reward as possible.
The deﬁnition of reward and constraint vary from task to task and will be explained when each task is introduced.
Our implementation will be made available upon acceptance of the paper.
Baseline Approaches.
Three CRL baselines are compared.
PCPO (Yang et al., 2020)
ﬁrst performs an unconstrained policy update then project the action to the constrained set.
CPO (Achiam et al., 2017) maximizes the reward in a small neighbourhood that enforces the constraints.
RCPO (Tessler et al., 2019) incorporates the cost terms and Lagrange multipliers with the reward function to encourage the satisfaction of the constraints.
We used the original implementation of
CPO and PCPO with KL-projection that leads to the best performance.
For RCPO, since the ofﬁcial implementation is not available, we re-implemented RCPO and made sure it matches the original performance.
RCPO restricts the expectation of the constraint values to be smaller than a threshold α instead of enforcing the constraints ρ max ( s ) and ρ min ( s ) for every state s .
All the three baseline approaches and our DCRL have the same number of neural network parameters.
Note that the baseline approaches enforce the constraints by restricting return values while our DCRL restricts the state densities .
The constraint thresholds of DCRL and the baseline methods can be inter-converted by the duality of density functions and value functions (Chen & Ames, 2019).
4. M U J O C O B ENCHMARK
Domain Description.
Experiments are conducted on three tasks adopted from CPO (Achiam et al., 2017) on the MuJoCo benchmark, including Point-Gather, Point-Circle and Ant-Circle.
In the PointGather task, a point agent moves on a plane and tries to gather as many apples as possible while keeping the probability of gather bombs below the threshold.
In the Point-Circle task, a point agent tries to follow a circular path that maximizes the reward, while constraining the probability of exiting an given area.
The Ant-Circle task is similar to the Point-Circle task except that the agent is an ant instead of a point.
detailed conﬁgurations of the benchmarks can be found in Achiam et al.
Experimental Results.
Figure 2 demonstrates the performance of the four methods.
In general,
DCRL is able to achieve higher reward than other methods while satisfying the constraint thresholds.
In the Point-Gather and Point-Circle environments, all the four approaches exhibit stable performance with relatively small variances.
In the Ant-Circle environment, the variances of reward and constraint values are signiﬁcantly greater than that in Point environments, which is mainly due to the complexity of ant dynamics.
In Ant-Circle, after 600 iterations of policy updates, the constraint values of the four approaches converge to the neighbourhood of the threshold.
The reward of
DCRL falls behind PCPO in the ﬁrst 400 iterations of updates but outperforms PCPO thereafter.
4. A UTONOMOUS E LECTRICAL V EHICLE R OUTING B ENCHMARK
Domain Description.
Our second case study is about controlling autonomous electric vehicles (EV) in the middle of Manhattan, New York.
It is adopted from Blahoudek et al.
and is shown in Figure 4.2.
When EVs drive to their destinations, they can avoid running out of power by recharging at the fast charging stations along the roads.
At the same time, the vehicles should not stay at the charging stations for too long in order to save resources and avoid congestion.
An road intersection is called a node.
In each episode, an autonomous EV starts from a random node and drives to the goals.
At each node, the EV chooses a direction and reaches the next node along that direction at the next step.
The consumed electric energy is assumed to be proportional to traveling distance.
The EV is fully charged at the beginning of each episode, and can choose to recharge at the fast charging stations.
There are 1024 nodes and 137 charging stations in total.
Denote the full energy capacity of the vehicle as c f and the remaining energy as c r .
When arriving at a charging station, the EV chooses a charging time τ ∈ [0 , 1] , then its energy increases to min( c f ,
c r + τc f ) .
The action space includes the EV direction and the charging time τ .
The state space S ⊂ R 3 is consisted of the current 2D location and
the remaining energy c r .
[Figure]
+ −
[Figure] [Figure]
Two types of constraints are considered: (1) the minimum remaining energy should keep close to a required threshold and (2) the vehicle density at charging stations should be less than a given threshold.
Apparently, if the EV chooses a larger τ , then the constraint (1) is more likely to be satisﬁed, while (2) is more likely to be violated, since a larger τ will increase the vehicle density at the charging station.
These contradictory constraints pose a greater challenge to the RL algorithms.
Both constraints can be naturally expressed as density constraints.
For constraint (1), we can limit the density of low-energy states.
For constraint (2), it is straightforward to limit the EV density (a function of E [ τ ] ) at charging stations.
The threshold of density constraints are transformed to the threshold of value functions to be used by the baseline methods.
The conversion is based on the duality of density functions and value functions (Chen & Ames, 2019).
Experimental Results.
Figure 3 demonstrates the performance of the four approaches.
Since this task has two contradicting constraints, it is challenging to satisfy both and at the same time maximize the reward.
Although the baseline methods can approximately satisfy both constraints, their reward values are lower than the proposed DCRL method.
Our
Proposition 1 reveals that DCRL can achieve the optimal reward when enforcing the constraints, which is an important reason why DCRL shows a higher reward than other methods in this task.
C ONCLUSION AND FUTURE WORKS
We introduced the DCRL problem to solve the RL problem while respecting the constraints on the state densities.
State densities have clear physical meanings and can express a variety of constraints of the environment and the system.
We proposed a model-free primal-dual algorithm to solve DCRL, which avoids the challenging problem of designing cost functions to encode constraints.
Note that our algorithm does not guarantee the density constraints are satisﬁed in the learning process.
In the future, we aim to improve the algorithm to enforce the density constraints during training as well.
We also plan to identify the assumptions to prove the convergence of our algorithm.
A P ROOFS OF STATEMENTS AND THEOREMS
In this section, the proofs omitted in our main paper are provided.
We will prove the equivalent expression of the state density function, the duality between density function optimization and Q function optimization in the unconstrained case (Theorem 1) and constrained case (Theorem 2), as well as the optimality of the proposed primal-dual algorithm for DCRL (Proposition 1).
A. E QUIVALENT EXPRESSION OF THE STATE DENSITY FUNCTION
In Section 3. [Equation] we point out that the state density function ρ πs has two equivalent expressions.
We start from Equation equation 1 and derive another expression:
[Equation]
A. P ROOF OF T HEOREM
The Lagrangian of (3) is
[Equation]
where µ : S ×
A → R is the Lagrange multiplier.
The key step is by noting that
[Equation]
Then by rearranging terms, the Lagrangian becomes
[Equation]
By the KKT condition and taking Q = µ , the optimality condition satisﬁes equation 4 exactly, and when optimality condition is attained, J (cid:63)d = L
= J (cid:63)p .
A. P ROOF OF T HEOREM
The solution π to the primal problem is the optimal policy for the modiﬁed MDP with reward r + σ − − σ + , which means π is the optimal solution to:
[Equation]
[Figure]
According to Theorem 1, π is also the optimal solution to:
[Equation]
Therefore, for any feasible policy π (cid:48)
the following inequality holds:
[Equation]
By complementary slackness, if σ − ( s ) > 0 , then ρ πs ( s ) = ρ min ( s ) .
The same applies to σ + and ρ max .
Since ρ π (cid:48) s ( s ) ≥ ρ min ( s ) and ρ π (cid:48) s [Equation] s ) ≤ ρ max ( s ) , we have:
[Equation]
Then we use equation 15 to eliminate the σ − ( s ) − σ + ( s ) in (14) and derive: [Equation]
[Equation]
which means π is the optimal solution maximizing J (cid:63)d among all the solutions satisfying the density constraints.
As a result, π is the optimal solution to the DCRL problem.
A. P ROOF OF P ROPOSITION
Note that when Algorithm 1 converges and the KKT conditions are satisﬁed, the policy that Algorithm 1 founds is the optimal solution to the primal problem (6) because the algorithm explicitly solves (6) in Line 6.
Thus, by Theorem 2, the policy is the optimal solution to the DCRL problem.
B S UPPLEMENTARY EXPERIMENTS
In this section, we provide new case studies that are not covered in the main paper.
We mainly compare with RCPO (Tessler et al., 2019) and the unconstrained DDPG (Lillicrap et al., 2016), which serves as the upper bound of the reward that can be achieved if the constraints are ignored. [Figure]
B. S AFE ELECTRICAL MOTOR CONTROL
The safe electrical motor control environment is from Traue et al.
and is illstrated in Figure 5 (a).
The objective is to control the motor so that its states follow a reference trajectory and at the same time to prevent the motor from overheating.
The state space S ⊂ R 6 and consists of six variables: angular velocity, torque, current, voltage, temperature and the reference angular velocity.
The action space A ⊂ R is the electrical duty cycle that controls the motor power.
The agent outputs a duty cycle at each time step to drive the angular velocity close to the reference.
When the reference angular velocity increases, the required duty cycle will need to increase.
As a result, the motor’s power and angular velocity will increase and cause the motor temperature to grow.
The algorithms are trained and tested using sawtooth-wave reference trajectories.
Results for other types of trajectories are presented in the supplementary material.
Figure 6 shows that our approach successfully controls the temperature density below the given threshold.
Unconstrained DDPG does not consider the temperature and only minimizes the difference between reference and measured angular velocity.
Therefore, the motor controlled using unconstrained DDPG has a signiﬁcant violation of the required temperature density when the motor temperature is high.
RCPO manages to avoid the high temperature but fails to suppress the density completely below the threshold.
As a comparison, our method manages to successfully control the temperature density completely below the required threshold.
To gain some insight on the different performance of the three methods, we visualize the trajectories and actions (duty cycles) taken at different temperatures and reference angular velocities in Figure 7.
In Figure 7 (a), The trajectory using DCRL can be divided into three phases.
In Phase 1, as the reference angular velocity grows, the duty cycle also increases, so the motor temperature goes up.
When the temperature is too high, the algorithm enters Phase 2 where it reduces the duty cycle to control the temperature, even though the reference angular velocity remains high.
As the temperature goes down, the algorithm enters Phase 3 and increases the duty cycle again to drive the motor angular velocity closer to the reference.
In Figure 7 (b), when the temperature is high, the RCPO algorithm will stop increasing the duty cycle but will not decrease it as Algorithm 1 does.
So the temperature remains high and thus the density constraints are violated.
In Figure 7 (c), the unconstrained DDPG algorithm continues to increase the duty cycle in spite of the high temperature.
To examine the generalization capability of our method, here we train on the sawtooth-wave trajectories but test on asymmetry sine trajectories that were not seen by Algorithm 1 before (Figure 8).
To make the experiments more diverse we also train and test the methods using staircase-wave trajectories (Figure 9).
Note that all the conﬁgurations of our DCRL method are the same in the three experiments (Figure 6, 8-9) and we did not tune any of the hyper-parameters for each type of reference trajectory.
In order to demonstrate the difﬁculty of tuning cost functions to satisfy the constraints with the
RCPO method we present the results of RCPO with different hyper-parameter conﬁgurations of the cost function (Figure 10).
Throughout the experiment, RCPO uses the cost function max { 0 , ( s ↓ t ) − η } , where s ↓ t is the temperature variable under state s .
The cost function is positive when s ↓ t exceeds η . η = 0 .
20 for RCPO, η = 0 .
23 for RCPO-V2, η = 0 .
17 for RCPO-V3 and η = 0 .
16 for RCPO-V4.
For fair comparisons, other parameters of these three variants of RCPO remain unchanged.
It is clear that RCPO-V4 is only slightly different from RCPO-V3.
As is shown in Figure 8, even if the testing reference trajectories are unseen, our approach still manages to control the temperature density below the threshold.
The sawtooth-wave trajectories are piece-wise linear, which are for training.
By contrast, the asymmetric sine-wave trajectories are nonlinear curves, which are for testing.
Figure 8 shows the generalization capability from linear to nonlinear reference trajectories without any re-training.
In Figure 9, while the angular velocity trajectory of RCPO shows unstable high-frequency oscillation, DCRL can stabilize the angular velocity and keep the temperature density completely below the threshold.
Unconstrained DDPG does not take into account the temperature density constraints and thus causes the motor to overheat.
All these scenarios show that our method can constantly outperforms RCPO and the unconstrained
DDPG in terms of guaranteeing satisfaction of constraints and generalization capability.
Different conﬁgurations in the cost (negative reward) function of RCPO will yield different results, and we show that it could be difﬁcult to tune the cost functions to satisfy the density constraints.
In
Figure 10, though it is still possible for RCPO to satisfy the density constraints by tuning the hyperparameters of the cost function (as RCPO-V3), a small perturbation would have an unexpected negative effect.
The hyper-parameter of RCPO-V4 is only slightly different from that of RCPO-V3, but RCPO-V4 completely fails to follow the reference trajectory.
A larger η will incur a heavier punishment for high temperature states, but tuning η could be difﬁcult.
Moreover, when the reference trajectory changes, the feasible η also changes, making the tuning process time-consuming.
In comparison, we did not tune any parameter of the DCRL method for each reference trajectory, which indicates the robustness and generalization capability of DCRL when applied to new scenarios.
B. A GRICULTURAL SPRAYING DRONE
We consider the problem of using drones to spray pesticides over a farmland in simulation.
Consider the area in Figure 5 (b).
The drone starts from the top-left corner, ﬂies over the farmland spraying pesticides, and stops at the bottom-right corner.
At each timestep, the agent outputs an acceleration parallel to the ground and then the drone moves horizontally at a ﬁxed height.
The drone has a constant spraying speed, which means when the drone moves faster, there would be less pesticides on the land that the drone passes over.
At each moment, the pesticides sprayed by the drone uniformly cover an area centered at the drone with diameter equals to 0.02 times the side length of the farmland.
Two constraints are (1) the pesticide density of each part of the land is within the predeﬁned minimum and maximum pesticide density requirements, and (2) the density (distribution) of drone velocity is below a predeﬁned threshold, which is to prevent dangerous high-speed movement.
As is shown in Figure 11, for each method we evaluate the area that satisﬁes pesticide density constraints, the time consumption and the velocity density.
While RCPO and our method demonstrate similar performance in terms of controlling the pesticide density and velocity density, our method requires fewer time steps to ﬁnish the task.
The unconstrained DDPG algorithm takes less time, but cannot satisfy the requirement on pesticide density or velocity density.
More experiments on the agricultural spraying problem are presented in the supplementary material.
We also examine the methods with different pesticide density requirements to assess their capability of generalizing to new scenarios.
For the farmland shown in Figure 5 (b), from area 0 to 4, the minimum and maximum pesticide density are (1 , 0 , 0 , 1 , 1) and (2 , 0 , 0 , 2 , 2) respectively.
In this supplementary material, we evaluate with two new conﬁgurations.
In Figure 12, the minimum and maximum density are set to (0 , 1 , 1 , 0 , 1) and (0 , 2 , 2 , 0 , 2) from area 0 to 4 .
In Figure 13, the
In Figure 12 and DCRL and RCPO demonstrates similar performance in controlling pesticide densities to be within the minimum and maximum thresholds, while DCRL demands less time to ﬁnish the task.
DDPG only minimizes the time consumption and thus requires the least time among the three methods, but cannot guarantee the pesticide density is satisﬁed.
In terms of the velocity control, both DCRL and RCPO can avoid the high-speed movement.
These observations suggest that when both DCRL and RCPO ﬁnds feasible policies satisfying density constraints, the policy found by DCRL can achieve lower cost or higher reward deﬁned by the original unconstrained problem, which is the time consumption of executing the task in this case study.
B. E XPRESS DELIVERY SERVICE TRANSPORTATION
An express delivery service company has several service points and a ship center in a city.
An example conﬁguration is illustrated in Figure 14 (a).
The company uses vans to transport the packages from each service point to the ship center.
The vans start from some service points following an initial distribution, travel through some service points and ﬁnally reach the ship center.
The cost is [Figure] [Table] formulated as the traveling distance.
The frequency that each service point is visited by vans should exceed a given threshold in order to transport the packages in the service points to the ship center.
Such frequency constraints can be naturally viewed as density constraints.
A policy is represented as the transition probability of the vans from one point to surrounding points.
The optimal policy should satisfy the density constraints and minimize the transportation distance.
Instead of comparing to methods that exert constraints on cost or value functions in the three case studies in Section 4, this case study is proposed to further understand Algorithm 1 and its key steps.
In Algorithm 1 Line 5, our approach adds Lagrange multipliers to the original reward in order to compute a policy that satisﬁes density constraints.
The update of Lagrange multipliers follows the dual ascent in Algorithm 1 Line 9 and 10, which is key to satisfying the KKT conditions.
In this experiment, we try to update the Lagrange multipliers using an alternative approach and see how the performance changes.
We replace the dual ascent with the cross-entropy method, where a set of Lagrange multipliers Σ =
[ σ 1 , σ 2 , σ 3 , · · · ] are drawn from an initial distribution σ ∼ Z ( σ ) and utilized to adjust the reward respectively, after which a set of policies [ π 1 , π 2 , π 3 , · · · ] are obtained following the same procedure in Algorithm 1 Line 5 and 6.
A small subset of Σ whose π has the least violation of the density constraints are chosen to compute a new distribution Z ( σ ) , which is utilized to sample a new Σ .
The loop continues until we ﬁnd a σ whose π completely satisﬁes the density constraints.
We call this cross-entropy reward shaping (CERS).
We experiment with 10D, 20D and 100D state spaces (corresponding to 10, 20 and 100 service points in the road network), whose density constraints lie in R 10 , R 20 and R 100 respectively.
The density constraint vector ρ min : S (cid:55)→ R is set to identical values for each state (service point).
For example, ρ min =
.indicates the minimum allowed density at each state is 0 .
In Algorithm 1 Line 6, we use QLearning to update the policy for both DCRL and CERS since the state and action space are discrete.
From Table 1, there are two important observations.
First, our computational time of ﬁnding the policy is signiﬁcantly less than that of CERS.
When ρ s ∈ R 10 and ρ min = 0 .
1 , our approach is at least 100 times faster than CERS on the same machine.
When ρ s ∈ R 100 and ρ min = 0 .
CERS cannot solve the problem (no policy found can completely satisfy the constraints) in the maximum allowed time (600s), while our approach can solve the problem in 153.86s.
Second, the cost reached by our method is generally lower than that of CERS, which means our method can ﬁnd better solutions in most cases.
[Figure]
All vehicles should maintain energy levels while avoiding congesting the charge stations.
B. A UTONOMOUS ELECTRIC VEHICLE ROUTING
In the autonomous electric vehicle routing task, we evaluate the methods in two new settings in addition to the one presented in our main paper.
As is shown in Figure 15, the goal sets (red nodes) are different from Figure 4.2.
The number of nodes and fast charging stations remain the same.
In Figure 16, three approaches exhibit similar performance in maintaining energy levels and avoiding running into low-energy states.
Nevertheless, only DCRL succeeds in preventing the congestion in the fast charging stations by suppressing the vehicle density at each fast charging stations below the density constraints.
RCPO is better than the unconstrained DDPG in terms of reducing the vehicle density, but still cannot keep the density completely below the thresholds.
In Figure 17, DCRL and RCPO demonstrates similar performance in avoiding low-energy states but DDPG violates the density constraints on low-energy states.
This is mostly because DDPG chooses a relatively short charging time in this scenario, which is consistent with the fact that DDPG only slightly violates the constraints on vehicle density at fast charging stations.
As for the time consumption to ﬁnish the task,
DCRL, RCPO and DDPG take 64.32, 60.89 and 58.40 steps on average to reach the goals shown in the left of Figure 15 and 56.31, 59.47 and 54.19 steps on average to reach the goals shown in the right of Figure 15.
DDPG minimizes the traveling distance and thus requires the least amount of traveling time, while DCRL shows the best performance in satisfying the vehicle density constraints at fast charging stations.
B. S AFE G YM
In this section we experiment with the Safe Gym (Ray et al., 2019) and compare with CPO (Achiam et al., 2017) and PPO (Schulman et al., 2017) with Lagrangian (PPO-L).
Both the CPO and PPOL are implemented by Ray et al.
We consider three environments in Safe Gym, including
PointGoal, PointButton and PointPush.
In PointGoal, the point agent aims to navigate to a goal while avoiding hazards.
In PointButton, the point agent aims to press the goal button and avoid hazards.
The agent will be penalized if it presses the wrong button.
In PointPush, the agent is expected to push a box to the goal position and avoid hazards.
A detailed description of the environments can be found in Section 4 of Ray et al.
Results are presented in Figure 18.
In the PointGoal environment, all the three methods can satisfy the constraints, while DCRL can achieve a higher average return comparing to other methods.
In the PointButton environment, CPO fails to satisfy the constraint.
In the PointPush environment, it is hard for all the methods to satisfy the constraints because the pushing task is more difﬁcult than the goal reaching task and button task.
Overall, DCRL shows the highest reward and the least violation of the constraints among the three methods.
C D ISCUSSION ON K ERNEL D ENSITY E STIMATION
[Figure]
We used the Epanechnikov kernel (visualized in (a)) and the linear kernel (visualized in (c)).
We estimate the state density w.r.t.
the horizontal distance to origin in the Point-Circle environment that is used in Section 4.
The densities are estimated with 1000 samples.
In Section 3.3, we presented an effective way to obtain the density function using kernel density estimation when the states are continuous.
In this section, we present both theoretical analysis of kernel density estimation and experimental result with different kernels and bandwidth and evaluate their inﬂuence on density estimation.
There exist many results about the accuracy of kernel density estimation, we quote the following lemma from (Wasserman, 2019):
Lemma 1.
Let h
=
[ h, ..., h ] be the bandwidth of the kernel with dimension d .
Let ˆ
ρ s be the kernel density estimation of ρ s , then for a ρ s ∈ Σ( β, L ) , for any ﬁxed δ > 0 ,
[Equation]
for some constants c and C where C depends on δ .
[Equation]
where D s g ( x )
=
∂ s
+ ... + sd ∂x s 11 ...∂x sdd and | s | = (cid:80) di
=1 s i .
This is adapted from Theorem 9 in (Wasserman, 2019), see more detail and other bias analysis results therein.
Lemma 1 essentially says that if ρ s is smooth enough and one takes enough samples, the bias of the kernel density estimator can be bounded uniformly in the state space.
We used the Epanechnikov kernel and linear kernel with various bandwidths to estimate the state density w.r.t.
the horizontal distance to origin in the Point-Circle environment that is used in Section 4.
Results are shown in Figure 19.
Although the kernels and bandwidths are different (see
Figure 19 (a) and (c)), the estimated densities (see Figure 19 (b) and (d)) are all close to the true densities (grey curves).
Once the upper bound (cid:15) ρ of the density estimation error is computed from Lemma 1, we can incorporate (cid:15) ρ into Algorithm 1 and derive Algorithm 2, which is robust under the possibly inaccurate density estimation.
In Line 9-10, we bloat the density thresholds ρ max and ρ min to ρ max − (cid:15) ρ and ρ min + (cid:15) ρ , which guarantee that the density constraints will still be satisﬁed even when there are inaccuracies in density function estimation.
[-ρ πs − ρ max + (cid:15) ρ ) = 0 , σ − · ( ρ min − ρ πs + (cid:15) ρ ) = 
-]