B ENCHMARKING C ONSTRAINT I NFERENCE IN I NVERSE R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
When deploying Reinforcement Learning (RL) agents into a physical system, we must ensure that these agents are well aware of the underlying constraints.
In many real-world problems, however, the constraints followed by expert agents (e.g., humans) are often hard to specify mathematically and unknown to the RL agents.
To tackle these issues, Inverse Constrained Reinforcement Learning (ICRL) considers the formalism of Constrained Markov Decision Processes (CMDPs) and estimates constraints from expert demonstrations by learning a constraint function.
As an emerging research topic, ICRL does not have common benchmarks, and previous works tested their algorithms with hand-crafted environments (e.g., grid worlds).
In this paper, we construct an ICRL benchmark in the context of two major application domains: robot control and autonomous driving.
For each environment, we design relevant constraints, generate the corresponding expert trajectories, and empirically justify the importance of these constraints.
To recover the constraints from expert demonstrations, previous ICRL methods typically learn a deterministic constraint function, which might dismiss the true constraint during training.
We tackle this issue by proposing a variational Bayesian approach to model the posterior distribution of candidate constraints.
Empirical evaluation shows this method outperforms other baselines in terms of collecting rewards and satisfying constraints.
The benchmark, including the instructions for reproducing
ICRL algorithms, is available at temporarily hidden due to the anonymous policy .
I NTRODUCTION
Constrained Reinforcement Learning (CRL) algorithms typically learn a policy under some known or predefined constraints (Liu et al., 2021)..
This setting, however, is not realistic in many real-world problems since it is difficult to specify the exact constraints that an agent should follow, especially when these constraints are time-varying, context-dependent and inherent to experts’ own experience, and further, such information may not be completely revealed to the agent.
For example, human drivers tend to determine an implicit speed limit and a minimum gap to other cars based on the traffic conditions, rules of the road, weather and social norms.
To derive a driving policy that matches human performance, an autonomous agent needs to infer these constraints from expert demonstrations.
An important approach to recovering the underlying constraints is Inverse Constrained Reinforcement Learning (ICRL)
Malik et al.
ICRL infers a constraint function (or sets for discrete space) to approximate constraints respected by expert demonstrations.
This is often done by alternating between updating an imitating policy and a constraint function.
Figure 1 summarizes the main procedure of ICRL.
As an emerging research topic, ICRL does not have common datasets and benchmarks for evaluation.
Existing validation methods heavily depend on the safe-Gym Ray et al.
and grid-world environments.
Utilizing these environments has some important drawbacks: 1) These environments are designed for control instead of constraint inference.
To fill this gap, previous works often pick some environments and add hand-crafted constraints to them.
Striving for simplicity, many of the selected environments have discretized state and action spaces of limited size Scobee & Sastry (2020); McPherson et al.
; Glazier et al.
(2021); Papadimitriou et al.
; Gaurav et al.
Generalizing model performance in these simple environments to practical applications will be difficult.
These environments do not include expert demonstrations, and thus previous works manually generate their own trajectories.
Since the quality of expert trajectories can significantly influence model performance, it is difficult to compare different ICRL algorithms without a consistent expert dataset.
In this paper, we propose a benchmark for evaluating Inverse Constrained Reinforcement Learning (ICRL) algorithms.
This benchmark includes both virtual environments and realistic environments.
The virtual environments are based on MuJoCo Todorov et al.
, but we update some of these robot control tasks by modifying reward functions and including specific constraints.
The realistic environments are constructed based on a highway vehicle tracking dataset Krajewski et al.
, so the environments can suitably reflect what happens in the realistic driving scenario.
We consider constraints about car velocities and distances in these realistic environments.
Each of these environments is accompanied by a demonstration dataset generated by an expert agent knowing the ground-truth constraints.
To demonstrate that the added constraints are important for control, we empirically compare the performance of agents trained with and without the constraints.
For ease of comparison, our benchmark includes the state-of-the-art ICRL baselines.
These baselines learn a deterministic constraint function that is most likely to differentiate expert trajectories (from
D e in Figure 1) from the generated ones (from D π i ) under some popular framework (e.g., Maximum
Entropy Ziebart et al.
(2008)), which, however, is often at the cost of dismissing the true constraint for temporally not possessing the most desirable qualities Chan & van der Schaar (2021).
On the other hand, a more conceptionally-satisfying method is accounting for all possibilities of the learned constraint by modeling its posterior distribution.
To extend this Bayesian approach to solve the task in our benchmark, we propose a Variational Inverse Constrained Reinforcement Learning (VICRL) algorithm that can efficiently infer constraints from the environment with a high-dimensional and continuous state space.
We demonstrate the performance of VICRL by comparing its rewards and constraint violation rate with other ICRL baselines in our virtual and realistic environments.
Empirical results show that the policies learned with the VICRL constraints can outperform others in terms of constraint satisfaction and reward accumulation.
R ELATED W ORK
Inferring Constraints from Demonstrations.
Previous works commonly inferred constraints to identify whether an action is allowed or a state is safe.
Among these works, (Chou et al., 2018; Scobee & Sastry, 2020; McPherson et al., 2021; Park et al., 2019) are based on the discrete state-action space and constructed constraint sets to distinguish feasible state-action pairs from infeasible ones.
Regarding continuous domains, the goal is to infer the boundaries between feasible and infeasible state-action pairs: (Lin et al., 2015; 2017; Armesto et al., 2017) estimated a constraint matrix from observations based on the projection of its null-space matrix.
(P ´ erez-D’Arpino & Shah, 2017) learned geometric constraints by constructing a knowledge base from demonstration.
(Menner et al., 2021) proposed to construct constraint sets that correspond to the convex hull of all observed data.
(Malik et al., 2021; Gaurav et al., 2022) approximated constraints by learning neural functions from demonstrations.
Some previous works (Calinon & Billard, 2008; Ye & Alterovitz, 2011;
Pais et al., 2013; Li & Berenson, 2016;
Mehr et al., 2016) focused on learning local trajectory-based constraints from a single trajectory.
These works focus on inferring deterministic constraints while some recent works have learned stochastic constraints, for example, (Glazier et al., 2021)
learned probabilistic constraints by assuming the environment constraint follows a logistic distribution.
(Chou et al., 2020;
Papadimitriou et al., 2021)
utilized a Bayesian approach to update their belief over constraints, but these methods are restricted to discrete state spaces or toy environments like grid-world.
Testing Environments for ICRL.
To the best of our knowledge, there is no common benchmark for ICRL, and thus previous works often define their own environments for evaluation, including
1) Grid-World is the most popular environment
due to its simplicity and interpretability.
Previous works (Scobee & Sastry, 2020; McPherson et al., 2021; Papadimitriou et al., 2021; Glazier et al., 2021;
Gaurav et al., 2022) added some obstacles to the grid map and examined whether their algorithms can locate these obstacles by observing expert demonstrations, but it is difficult to generalize the model performance in these gird worlds to real applications with high-dimensional and continuous state spaces.
2) Robotic Applications have been used as the test bed for constraint inference, for example, the manipulation of robot arms (Park et al., 2019; Menner et al., 2021;
Armesto et al., 2017;
P´erez-D’Arpino & Shah, 2017), quadrotors (Chou et al., 2019; 2020), and humanoid robot hands (Lin et al., 2017).
However, there is no consistent type of robot for comparison, and the corresponding equipment is not commonly available.
A recent work (Malik et al., 2021) used a robotic simulator by adding some pre-defined constraints into the simulated environments.
Our virtual environments use a similar setting, but we cover more control tasks and include a detailed study of the environments and the added constraints.
3) Safety-Gym (Ray et al., 2019) is one of the most similar benchmarks to our work.
However, Safety Gym is designed for validating the forward policy-updating algorithm given the constraints, whereas our benchmark is designed for the inverse constraint-inference problem.
B ACKGROUND
In this section, we introduce Inverse Constrained Reinforcement Learning (ICRL) that learns constraint functions by iteratively solving both a forward Constrained Reinforcement Learning problem (CRL) and an inverse constraint inference problem (see Figure 1).
3. C ONSTRAINED R EINFORCEMENT L EARNING
Constrained Reinforcement Learning (CRL) is based on Constrained Markov Decision Processes (CMDPs)
M c , which can be defined by a tuple ( S , A , p R , p T , { ( p C i , ϵ i ) } ∀ i , γ, T ) where: 1) S and
A denote the space of states and actions.
2) p T ( s ′ | s, a ) and p R ( r | s, a ) define the transition and reward distributions. 3)
p C
i ( c | s, a ) denotes a stochastic constraint function with an associated bound ϵ
i , where i indicates the index of a constraint. 4) γ ∈ [0 , 1) is the discount factor and h is the planning horizon (usually infinite).
The goal of CRL is to find a (deterministic or stochastic) policy π that maximizes expected discounted rewards under a set of cumulative soft constraints:
[Equation]
In the above formulation, constraints consist of bounds on the expectation of cumulative constraint values.
In practice, it is often more desirable to express bounds on the cumulative constraint value of a single trajectory instead of the expectation of all trajectories.
Hence, we consider a more restrictive objective that requires all the sampled trajectories to satisfy the constraints:
[Equation]
where c i,t ∼ p C
i ( c | s t , a t ) and ( s t , a t ) ∼ p T ( s t | s t − 1 , a t − 1 ) π ( a t
| s t ) .
This formulation effectively describes hard constraints for each trajectory that may be difficult to satisfy in stochastic environments.
Nevertheless, hard constraints are natural and often desired by practitioners.
Hence, we will report the constraint violation rate (i.e., the probability with which a policy violates a constraint in a trajectory) in the experiments (see Section 4 and Section 6).
Ultimately, we can think of this setup as a multiobjective problem where agents seek to maximize expected rewards while minimizing the constraint violation rate.
Note that CRL problems often assume the constraint signals are directly observable from the environment, but in real-world problems, such constraint signals are not readily available, and we must infer these constraints from the environment by solving the following inverse problem.
3. I NVERSE C ONSTRAINT I NFERENCE
In practice, instead of observing the constraint signals, we often have access to expert demonstrations that follow the underlying constraints.
Under this setting, the agent must recover the constraint models from the dataset.
This is a challenging task since there might be various equivalent combinations of reward distributions and constraints that can explain the same expert demonstrations (Ziebart et al., 2008; Malik et al., 2021).
To simplify the problem, ICRL algorithms generally assume that rewards are observable and the goal is to recover only the constraints.
This is the key difference with Inverse
Reinforcement Learning (IRL) whose goal is to learn rewards from an unconstrained MDP.
Although the goals are different, ICRL and IRL both solve the problems of learning from demonstrations and thus have many connections, for example, our baselines cover previous works (Scobee & Sastry, 2020) that utilized the Maximum Entropy IRL framework for constraint inference.
Besides, IRL algorithms often assume that the expert agents are optimal or near-optimal in terms of maximizing the rewards.
ICRL further assumes that the agents must satisfy underlying constraints.
How to relax these assumptions will be an important direction of future work.
B ENCHMARK
The goal of ICRL is to discover constraints from demonstration data.
However, the true constraints in real-world datasets are often unknown, which makes it difficult to quantify the quality of learned constraints.
To solve these issues, we develop an ICRL benchmark that incorporates external constraints to different environments.
This benchmark facilitates the development of ICRL algorithms by 1) providing both virtual and realistic environments for evaluation and 2) enabling the quantification of model performance with some common RL metrics.
For ease of application, we introduce some predefined constraints (see Table 1), generate the corresponding expert datasets, and empirically demonstrate their effectiveness under these environments.
4. V IRTUAL E NVIRONMENT
An important application of RL is robotic control, and our virtual benchmark mainly studies the robot control task with a location constraint.
In practice, this type of constraint captures the locations of obstacles in the environment.
For example, the agent observes that none of the expert agents visited some places in the environment.
Then it is reasonable to infer that these locations must be unsafe, which can be represented by constraints.
Although the real-world tasks might require more complicated constraints, our benchmark, as the first benchmark for ICRL, could serve as a stepping stone for these tasks.
Environment Settings.
We implement our virtual environments by utilizing MuJoCo Todorov et al.
(2012), a virtual simulator suited to robotic control tasks.
To extend MuJoCo for constraint inference, we modify the MuJoCo environments by incorporating some predefined constraints into each environment and adjusting some reward terms.
Table 1 summarizes the environment settings (see Appendix A.1 for more details).
The virtual environments have 5 different robotic control environments simulated by MuJoCo.
We add constraints on the X -coordinate of these robots: 1) For the environments where it is relatively easier for the robot to move backward rather than forward (e.g., Half-Cheetah, Ant, and Walker), our constraints bound the robot in the forward direction (with a large X -coordinate), 2) For the environments where moving forward is easier (e.g., Swimmer), the constraint bounds the robot in the backward direction (with a small X -coordinate).
In these environments, the rewards are determined by the distance that a robot moves between the current and the previous time step, so the robot is likely to violate the constraints in order to maximize the magnitude of total rewards (for which we show in Section 4.4).
To increase difficulty, we include an additional Biased Pendulum environment that has a larger reward on the left side.
We nevertheless enforce a constraint to prevent the agent to go too far on the left side.
The agent must resist the influence of high rewards and stay in safe regions.
[Table]
Note that our virtual benchmark has not included some popular environments with discrete state and action spaces (e.g., Gird-World), for which we clarify from the following perspectives: 1) Real-world
Application .
One of the main goals of proposing this benchmark is to fill the gap between the simulated environment with the real-world application (see our introduction), and generalizing the performance of the discrete benchmark to real-world applications in continuous domain input is difficult. 2) Optimization Goals.
Under the discrete environments (e.g., Grid-worlds), Bellman optimality can be reached by dynamic programming (an optimal policy ( π ∗ = arg max a Q ∗ ( s, a ) ) can be uniquely found)
Sutton & Barto (2018), whereas, in this work, our baselines utilizes function approximation for tackling continuous domains.
These algorithms approximate the optimal policies by minimizing a sub-optimal upper bound (e.g., on the regret ∥ ˆ Q − Q ∗ ∥ ∞ ). There is no guarantee that our baseline can converge to the optimal policy satisfying the Bellman Optimality. The gap between continuous space and tractable discrete space is significant, which forms a tradition to evaluate the tabular-based algorithms with Grid-worlds Scobee & Sastry (2020) and the function approximation algorithms with continuous environments respectively Malik et al. 4. R EALISTIC E NVIRONMENT
Our realistic environment defines a highway driving task.
This HighD environment examines if the agent can drive safely the ego car to the destination by following the constraints learned from human drivers’ trajectories (see Figure 2).
In practice, many of these constraints are based on driving context and human experience.
For example, human drivers tend to keep larger distances from trucks and drive slower on crowded roads.
Adding these constraints to an auto-driving system can facilitate a more natural policy that resembles human preferences.
Environment Settings.
This environment is constructed by utilizing the HighD dataset Krajewski et al.
Within each recording, HighD contains information about the static background (e.g., the shape and the length of highways), the vehicles, and their trajectories.
We break these recordings into 3,041 scenarios so that each scenario contains less than 1,000 time steps.
To create the RL environment, we randomly select a scenario and an ego car for control in this scenario.
The game context, which is constructed by following the background and the trajectories of other vehicles, reflects the driving environment in real life.
To further imitate what autonomous vehicles can observe on the open road, we ensure the observed features in our environment are commonly used for autonomous driving (e.g., velocity, and distances to nearby vehicles).
These features reflect only partial information about the game context.
To collect these features, we utilize the features collector from Commonroad RL Wang et al.
In this HighD environment, we mainly study a car velocity constraint and a car distance constraint (see Table 1) to ensure the ego car can drive at a safe speed and keep a proper distance from other vehicles.
Our experiment in Section 6 also studies an environment having both of these constraints.
Note that the HighD environment is stochastic since 1) Human drivers might behave differently under the same context depending on the road conditions and their driving preferences.
The population of drivers induces underlying transition dynamics that are stochastic.
The trajectories in the HighD dataset are essentially samples from these stochastic transition dynamics.
Each time an environment is reset (either the game ends or the step limit is reached), it randomly picks a scenario with a set of driving trajectories.
This is equivalent to sampling from the aforementioned transition dynamics.
4. D EMONSTRATION D ATASET
In this work, we generate the expert demonstration dataset for constraint inference since 1) the virtual environments do not include demonstrations and 2) utilizing the trajectories in the HighD dataset under the realistic environments is problematic.
This is because the underlying constraints in human demonstrations are unknown, which makes it difficult to determine whether the ground-truth constraints are broken.
To generate the dataset, we train a PPO-Lagrange (PPO-Lag) under the CMDP with the known constraints (Table 1 and Appendix A.1) by performing the following steps:
Training Expert Agent with PPO-Lag.
We train expert agents based on ground-truth constraints in both virtual environments and realistic environments.
The expert agent is trained by maximizing the following PPO-Lag objective based on the Reward Constrained Policy Optimization (RCPO) method Tessler et al.
[Equation]
where H denotes the entropy term and c ∗ is the ground-truth constraint function from the environment.
Note that PPO-Lag assumes the reward and the constraint functions are deterministic and the agent can learn to avoid most of the constrained state-action pairs with the Lagrangian penalty term.
The empirical studies (Sections 4.4) show that PPO-Lag can achieve a satisfactory performance (although not optimal) given the ground-truth constraint function.
Generating a Dataset with Expert Agents.
We initialize D e = {∅} and run the trained expert agents in both the virtual and the realistic environments.
While running, we monitor whether the ground-truth constraints are violated until the game ends.
If not, we record the corresponding trajectory: D e = D e ∪ { τ e } , otherwise, we abandon this trajectory.
We repeat this process until the demonstration dataset has enough trajectories.
Note that PPO-Lag is **not** optimal in terms of maximizing the rewards or satisfying constraints.
After filtering the generated trajectories with state-action pairs that violate constraints, the recorded data D e must satisfy the added constraints.
However, there is no guarantee that the maximum number of rewards is collected in D e .
For more details, please check Appendix C. Our experiment (Section 6) shows ICRL algorithms can outperform
PPO-Lag under some easier environments.
4. E MPIRICAL S TUDY ABOUT C ONSTRAINTS .
In this section, we empirically demonstration the validity of the constraints defined in Table 1.
Constraints in the Virtual Environments.
The thresholds of the constraints are determined experimentally to ensure that these constraints ”matter” for solving the control problems.
This is shown in
Figure 3: 1) without knowing the constraints, a PPO agent tends to violate these constraints in order to collect more rewards within a limited number of time steps.
2) When we inform the agent of the ground-truth constraints (with the Lagrange method in Section 4.3), the PPO-Lag agent learns how to stay in the safe region, but the scale of cumulative rewards is likely to be compromised.
Based on these observations, we can evaluate whether the ICRL algorithms have learned a satisfying constraint function by checking whether the corresponding RL agent can gather more rewards by performing feasible actions under the safe states (i.e., by checking rewards and the constraint violation rate ).
Constraints in the Realistic Environments.
We show the difference in performance between a PPO-Lag agent (Section 4.3) that knows the ground-truth constraints and a PPO agent without knowing the constraints.
Figure 4 reports the violation rate of the speed constraint (top left) and the distance constraint (top right).
The bottom graphs report the cumulative rewards in both settings.
We find 1) the PPO agent tends to violate the constraints in order to get more rewards and 2) the PPO-Lag agent abandons some of these rewards in order to satisfy the constraints.
Their gap demonstrates the significance of these constraints.
Appendix A.5 explains why these constraints are ideal by comparing them with other candidate constraint thresholds.
V ARIATIONAL I NVERSE C ONSTRAINED R EINFORCEMENT L EARNING
In this section, we introduce our VICRL for constraint inference.
Unlike previous ICLR baselines that model a single constraint function, VICRL infers the distribution of candidate functions for better coverage of the true constraint.
To scale Bayesian inference to high-dimensional state space,
VICRL extends the variational Bayesian framework to infer constraints from the expert demonstration.
The goal of VICRL is to infer the distribution of a feasibility variable Φ so that p ( ϕ | s, a ) measures to what extent an action a should be allowed in a particular state s 1 .
The instance ϕ can define a soft constraint given by: ˆ c ϕ ( s, a ) =
1 − ϕ where ϕ ∼ p ( ·| s, a ) .
Given a CMDP M ˆ c ϕ based on the estimated constraint function ˆ c ϕ , we use p ( D e | ϕ ) to define the likelihood of generating the demonstration dataset D e .
Under this setting, the true posterior p ( ϕ |D e ) is intractable due to high-dimensional input space, so VICRL learns an approximate posterior q ( ϕ |D e ) by minimizing
D kl (cid:104) q ( ϕ |D e ) ∥ p ( ϕ |D e ) (cid:105) .
This is equivalent to maximizing an Evidence Lower Bound (ELBo):
[Equation]
We introduce our method by defining the key components in this objective as follows:
The Log- Likelihood Term
We define the log-likelihood log [ p ( D e | ϕ ) by utilizing the Maximum
Entropy IRL
method on a constrained MDP M ˆ c ϕ Malik et al.
; Ziebart et al.
(2008): log [ p ( D e | ϕ )]
= log (cid:34) (
Z M ˆ cϕ ) N
N (cid:89)
[Equation]
[Equation]
( τ ( i ) ) (cid:105) 1 M ˆ [Equation] [Equation] where 1) N denotes the number of trajectories in the demonstration dataset D e , 2) the normalizing term Z M ˆ cϕ
= (cid:82) exp
[ r ( τ )] 1 M ˆ cϕ ( τ )d τ , and 3) the trajectory identifier 1 M ˆ cϕ ( τ ( i ) ) can be defined by ϕ ( τ ( i ) )
= (cid:81)
Tt =1 ϕ t and ϕ t ∼ p ( ϕ | s it , a it ) , which defines to what extent the trajectory τ ( i ) is feasible.
Since Φ is a continuous variable with range [0 , 1] , we parameterize p ( ϕ | s, a ) by a Beta distribution: ϕ [Equation] s, a [Equation]
| [Equation]
, a [Equation] = Beta ( α, β ) where [ α, β ] = log[1 + exp( f ( s, a ))] (6) where f is implemented by a multi-layer network with 2-dimensional outputs (for α and β ).
For simplicity, we slightly overload the symbols by using (cid:81) Tt =1 ϕ
( s it , a it ) to denote the trajectory identifier 1 M c ( τ ( i ) ) and substitute it in Equation (5).
In this way, we can define:
[Equation]
The gradient of this likelihood function is given by:
[Equation]
So maximizing log [ p ( D e | ϕ )] is equivalent to maximizing the objective:
[Equation]
where ˆ τ is sampled based on executing policy π M ˆ ϕ (ˆ τ )
=
exp[ r ( τ )] ϕ (ˆ τ ) (cid:82) exp
[ r ( τ )] ϕ ( τ )d τ .
This is a maximum entropy policy that can maximize cumulative rewards subject to π M ϕ ( τ ) = 0 , ∀ (cid:80) ( s,a ) ∈
τ ˆ c ϕ ( s, a ) >
ϵ (note that ˆ c ϕ
( s, a ) = 1 − ϕ t as defined above).
We follow Tessler et al.
and learn this policy with a RCPO objective.
This is equivalent to substituting c ∗ with the estimated ˆ c ϕ in Equation (3).
The KL Divergence.
Striving for the ease of computing mini-batch gradients, we approximate
D kl (cid:104) q ( ϕ |D )
∥ p ( ϕ ) (cid:105) with (cid:80) ( s,a ) ∈D D kl (cid:104) q
( ϕ | s, a ) ∥ p ( ϕ ) (cid:105) .
Since both the posterior and the prior are
Beta distributed, we define the KL divergence by following the Dirichlet VAE Joo et al.
[Equation]
where 1) [ α 0 , β 0 ] and [ α, β ] are parameters from the prior and the posterior functions.
2) Γ and ψ denote the gamma and the digamma functions.
E MPIRICAL E VALUATION
Experiment Setting.
Following (Malik et al., 2021), we evaluate the quality of a recovered constraint by whether the corresponding imitation policy can maximize the cumulative rewards with a minimum violation rate for the ground-truth constraints.
We repeat each experiment with different random seeds, according to which we report the mean ± standard deviation (std) results for each studied baseline and environment.
For the details of model parameters and random seeds, please see Appendix A.2.
Baselines.
1) Maximum Entropy Constraint Learning (MECL) is based on the Maximum Entropy
IRL (Ziebart et al., 2008) framework, with which Scobee & Sastry (2020) proposed an algorithm to search for constraints that most increase the likelihood of observing expert demonstrations.
This algorithm focused on only discrete state spaces.
A following work (Malik et al., 2021) expanded
MECL to continuous states and actions, which is consistent with the setting of our benchmark. 2)
Generative Adversarial Constraint Learning (GACL) extends MECL by following the design of
Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon, 2016).
where ζ ( s, a ) assign 0s to violating state-action pairs and 1s to satisfying ones.
In order to include the learned constraints into the policy update, we follow (Malik et al., 2021) and construct a new reward r ′ ( s, a ) =
r ( · ) + log [ ζ ( · )] .
This reward will punish the violating states or actions by assigning them −∞ rewards, so the learned policy tends to satisfy the constraints.
3) Binary Classifier Constraint Learning (BC2L) shares a similar policy update method with MECL, but instead of learning a soft constraint, it utilizes a deterministic binary classifier to differentiate expert trajectories from the generated ones, which often induces a loss of identifiability.
Performance in Virtual Environments.
Figure 5 and Table 2 show the training curves and the corresponding testing performance in each virtual environment.
Compared to other baseline models, we find VICRL generally performs better with lower constraint violation rates and larger cumulative rewards.
This is because VICRL captures the uncertainty of constraints by modeling their distributions and requiring the agent to satisfy all the sampled constraints, which facilitates a conservative imitation policy.
Although MECL and GACL outperform VICRL in the Blocked Walker and the Blocked
Swimmer environments, respectively, none of these algorithms can perform consistently better than the others.
Figure B.2 visualizes the constraints learned by VICRL for a closer analyse.
Performance in Realistic Environments.
Figure 7 shows the training curves and Table 2 shows the testing performance.
Among the studied method, VICRL achieves a lower constraint violation rate with a satisfying number of rewards.
Although GACL has the lowest violation rate, it is at the cost of significantly degrading the controlling performance, which demonstrates that directly augmenting rewards with penalties (induced by constraints) can yield a sub-optimal control policy.
Appendix B. illustrates the causes of failures by showing the collision rate, time-out rate, and off-road rate.
To illustrate how well the constraint is captured by the experimented algorithms, our plots include the [Table] 100 runs. We repeat the test for the models trained by one random seed (we have a total of 5 random seeds) and report the averaged p values.
Blocked
[Table]
- Cheetah
BlockedAnt
BiasedPendulum
BlockedWalker
[Table]
HighDVelocity
HighDDistance
FeasibleRewards
[Table]
ConstraintViolationRate
[Table]
upper bound of rewards and the performance of the PPO-Lag agent (trained under the true constraints).
It shows that there is sufficient space for future improvement under our benchmark.
To better understand the influence of environments, we explore inferring constraints from:
Stochastic Dynamics.
We extend the MuJoCo environments by adding noise to the transition functions at each step such that p ( s t +1 | s t , a t ) =
f ( s t , a t ) + η, where η ∼ N ( µ, σ )) ,
Figure 6 shows the results.
We find
ICRL models are generally robust to additive Gaussian noises in environment dynamics until they reach a limit (e.g., N (0 , 0 . 1) ).
Another intriguing finding is that the deterministic constraint inference methods (MECL and B2CL) can benefit from a proper scale of random noise since these noisy signals facilitate a more restricted constraint function and thus a lower constraint violation rate (middle column in Figure 6).
Noisy Demonstration. We add noise to the expert demonstration dataset by performing random actions with a chance of 100%, 80%, 50%, 20% and 0% during data generation.
The results in
Figure 8 (left column) show the model performance, especially the rewards, is significantly influenced as we expand the scale of noise (from 0% to 100%).
This is because the random actions induce sub-optimal trajectories.
These trajectories are less effective in terms of reflecting the preference of expert agents, which degrades the quality of learned constraints.
Incomplete Demonstration.
We randomly remove samples from the expert demonstration dataset by keeping only 30%, 50%, and 100% of the total trajectories.
The results in Figure 8 (right column) show that the constraint violation rate becomes higher when we reduce the size of demonstration data.
This is because the expert agent commonly applies a stochastic policy, so the diversity of recorded data is influenced if we reduce the roll-out numbers (i.e., the number of trajectories).
Multi-Constraints Environment.
The previously studied environments include only a true constraint.
Now we study the environment that has both the speed and the distance constraints in the HighD environment.
Figure 9 shows the results.
Compared to the performance of its single-constraint counterparts (in Figure 7), the rewards collected by the imitation policy become significantly lower, although the constraint violation rate remains uninfluenced.
Our VICRL consistently outperforms other baselines in terms of collecting rewards and respecting constraints. Discussion.
We summarize the major challenges in our benchmark as follows: 1) Constraint Satisfaction.
In the experiment, we observe that the learned policy may not match the expert policy even when the ground-truth constraint is recovered.
This is because the Lagrange relaxation methods are sensitive to the initialization of the Lagrange multipliers and the learning rate.
There is no guarantee that the policies can consistently satisfy the given constraints(See Figure 4).
2) Noisy Demonstration Data. ICRL algorithms typically assume that the expert demonstrations are optimal in terms of satisfying the constraints and maximizing rewards. There is no guarantee that these assumptions hold in practice since many expert agents (e.g., humans) do not always strive for optimality and constraint satisfaction. Experiment results (Figure 8) demonstrate model performance significantly degrades datasets including noisy signals.
3) High-dimensional Features.
To learn the accurate constraint, the agent must identify exact features (e.g., velocity in x and y coordinates) among all input features (78 in total) and recover the exact constraint threshold (e.g., 40 m/s), which explains the limited model performance in the complex environment (e.g., HighD, Figure 7 and Figure 9).
C ONCLUSION
In this work, we introduced a benchmark, including robot control environments and highway driving environments, for evaluating ICRL algorithms.
Each environment is aligned with a demonstration dataset generated by expert agents.
To extend the Bayesian approach to constraint inference, we proposed VICRL to learn a distribution of constraints.
The empirical evaluation showed the performance of ICRL algorithms under our benchmark.
A M ORE I MPLEMENTATION AND E NVIRONMENT D ETAILS A. M ORE I NFORMATION ABOUT THE V IRTUAL E NVIRONMENTS
Our virtual environments are based on Mujoco (see Figure A.1).
We provide more details about the virtual environments as follows
: • Blocked Half-Cheetah.
The agent controls a robot with two legs.
The reward is determined by the distance it walks between the current and the previous time step and a penalty over the magnitude of the input action.
The game ends when a maximum time step (1000) is reached.
We define a constraint that blocks the region with X-coordinate ≤ − 3 , so the robot is only allowed to move in the region with X-coordinate between -3 and ∞ .
• Blocked Ant.
The agent controls a robot with four legs.
The rewards are determined by the distance to the origin and a healthy bonus that encourages the robot to stay balanced.
The game ends when a maximum time step (500) is reached.
Similar to the Blocked Half-Cheetah environment, we define a constraint that blocks the region with X-coordinate ≤ − 3 , so the robot is only allowed to move in the region with X-coordinate between -3 and ∞ .
• Biased Pendulum.
Similar to the Gym CartPole (Brockman et al., 2016), the agent’s goal is to balance a pole on a cart.
The game ends when the pole falls or a maximum time step (100) is reached.
At each step, the environment provides a reward of 0.1 if the X-coordinate ≥ 0 and a reward of 1 if the X-coordinate ≤ − 0 .
The reward monotonically increases from 0.1 towhen − 0 .
01 <
X-coordinate < 0 .
We define a constraint that blocks the region with X-coordinate ≤ − 0 .
015 , so the reward incentivizes the cart to move left, but the constraint prevents it from moving too far.
If the agent can detect the ground-truth constraint threshold, it will drive the cart to move into the region with X-coordinate between − 0 .
015 and − 0 .
01 and stay balanced there.
• Blocked Walker.
The agent controls a robot with two legs and learns how to make the robot walk.
The reward is determined by the distance it walks between the current and the previous time step and a penalty over the magnitude of the input action (this is following the original Walker2d environment).
The game ends when the robot loses its balance or reaches a maximum time step (500).
Similar to the Blocked Half-Cheetah and Blocked Ant environment, we constrain the region with X-coordinate ≤ − 3 , so the robot is only allowed to move in the region with X-coordinate between -3 and ∞ .
• Blocked Swimmer.
The agent controls a robot with two rotors (connecting three segments) and learns how to move.
The reward is determined by the distance it walks between the current and the previous time step and a penalty over the magnitude of the input action.
The game ends when the robot reaches a maximum time step (500).
Unlike the Blocked Half-Cheetah and Blocked Ant environment, it is easier for the Swimmer robot to move ahead than move back, and thus we constrain the region with X-coordinate ≥ 0 .
5 , so the robot is only allowed to move in the region with X-coordinate between −∞ and 0 .
A. H YPER -P ARAMETERS
We published our benchmarks, including the configurations of the environments and the models, in temporarily hidden due to the anonymous policy .
Please see the README.
MD file for more details.
We provide a brief summary of the hyper-parameters.
In order to develop a fair comparison among ICRL algorithms, we use the same setting for all algorithms.
In the virtual environments , we set 1) the batch size of PPO-Lag to 64, 2) the size of the hidden layer to 64, and 3) the number of hidden layers for the policy function, the value function, and the cost function to 3.
We decide the other parameters, including the learning rate of both PPO-Lag and constraint model, by following some previous work (Malik et al., 2021) and their implementation temporarily hidden due to the anonymous policy .
The random seeds of virtual environments are 123, 321, 456, 654, and 666.
In the realistic environments , we set 1) the batch size of the constraint model to 1000, 2) the size of the hidden layer to 64 and 3) the number of hidden layers for the policy function, the value function and the cost function to 3.
We decide the other parameters, including the learning rate of both PPO-Lag and constraint model, by following CommonRoad RL (Wang et al., 2021) and their implementation 2 .
During our experiment, we received plenty of help from their forum 3 .
We will acknowledge their help in the formal version of this paper.
The random seeds of realistic environments are 123, 321, and 666.
A. E XPERIMENTAL E QUIPMENT AND I NFRASTRUCTURES
We run the experiment on a cluster operated by the Slurm workload manager.
The cluster has multiple kinds of GPUs, including Tesla T4 with 16 GB memory, Tesla P100 with 12 GB memory, and RTX 6000 with 24 GB memory.
We used machines with 12 GB of memory for training the ICRL models.
The number of running nodes is 1, and the number of CPUs requested per task is 16.
Given the aforementioned resources, running one seed in the virtual environments and the realistic environments takes 2-4 hours and 10-12 hours respectively.
A. C OMPUTATIONAL C OMPLEXITY
We provide a brief analysis of the computational complexity.
The ICRL algorithms, including
GACL, MECL, BC2L, and VICRL, use an iterative updating paradigm and thus their computational complexities are similar.
Let K denote the number of iterations.
Within each iteration, the algorithms update both the imitation policy and the constraint model.
Let M denote the number of episodes that the PPO-Lag algorithm runs in the environments.
Let N denote the number of sampling and expert trajectories.
Let L denote the maximum length of each trajectory.
During training, we use mini-batch gradient descent.
Let B denote the batch size, and then the computational complexity is
O ( KL ( M + N ) /B ) .
A. E XPLORING O THER C ONSTRAINTS IN THE R EALISTIC E NVIRONMENTS
The constraint thresholds in our environments are determined empirically according to the performance (constraint violation rate and rewards) of the PPO agent and the PPO-Lag agent.
To support this claim, we show the performance of other thresholds and analyze why they are sub-optimal in terms of validating ICRL algorithms.
We have explored the option of using a 30m/s velocity constraint (The first column on the left in
Figure A.2) and 35m/s velocity constraint (The second column on the left in Figure A.2).
Ideally, these constraints should be closer to the realistic speed limit in most countries.
However, the HighD the dataset comes from German highways where there is no speed limit.
Moreover, when building the environment, the ego car is accompanied by an initial speed calculated from the dataset.
We observed that the initial speed is already higher than the speed limit (e.g., 35m/s) in many scenarios, and thus the violation rate will always be 1 in these scenarios, leaving no opportunity for improving the policy.
This explains why the corresponding violation rates are high for the PPO and the PPO-Lag agents.
We also explored the option of using a 40m distance constraint (third column in Figure A.2) and a 60m distance constraint (fourth column in Figure A.2).
Ideally, these constraints should be more consistent with the 2-second gap recommendation (the average speed is around 30m/s in HighD, so the recommended gap is 2*30m/s=60m)
[Table]
but we find the controlling performance of the PPO-Lag agents are very limited [Table] which shows the agent cannot even develop a satisfying control policy when knowing the ground-truth constraints.
This is because the ego car learns to frequently go off-road in order to maintain the large gap
[Table]
B M ORE E XPERIMENTAL R ESULTS B. T HE C OMPLETE R ESULTS FOR T ESTING P ERFORMANCE
[Table]
[Table] [Table]
[Table] [Table]
B. C OMPLEMENTARY R ESULTS IN THE R EALISTIC E NVIRONMENT
Figure B.1 reports the average velocity, collision rate, off-road rate, time-out rate and goal-reaching rate during training.
We find the off-road rate of GACL is significantly higher than other methods.
It explains why GACL cannot achieve a satisfying performance.
Another main limitation of current baselines is their incapability of preventing the collision events, especially under the car distance constraints.
[Table]
B. C ONSTRAINT V ISUALIZATION
accompanied by the samples from the constraint distribution (blue points, i.e., P C ( c | s, a ) , marked by cost) and 2) histograms (blue, on the right) showing the number of states with a specific feature value (e.g., x position) during testing.
The x-axis of these plots show the features where the ground-truth constraints are defined on (this message is hidden from the agents during training).
In order to understand how well the constraints are captured, we can compare these plots with the definition of ground truth constraints in Table 1.
C L IMITATIONS , C HALLENGES AND O PEN Q UESTIONS
We introduce limitations and challenges in ICRL, as well as open questions for future work.
Constraint Violation.
The imitation policies of ICRL agents are updated with RCPO (Tessler et al., 2019), but Lagrange relaxation methods are sensitive to the initialization of the Lagrange multipliers and the learning rate.
There is no guarantee that the imitation policies can consistently satisfy the given constraints (Liu et al., 2021).
As a result, even when a learned constraint function matches the ground-truth constraint, the learned policy may not match the expert policy, causing significant variation in training and sub-optimal model convergence.
If we replace the Lagrange relaxation with
Constrained Policy Optimization (CPO) (Achiam et al., 2017; Chow et al., 2019; Yang et al., 2020;
Liu et al., 2020), ICRL may not finish training within a reasonable amount of time since CPO is computationally more expensive.
How to design an efficient policy learning method that matches
ICRL’s iterative updating paradigm will be an important future direction.
Unrealistic Assumptions about Expert Demonstrations.
ICRL algorithms typically assume that the expert demonstrations are optimal in terms of satisfying the constraints and maximizing rewards.
There is no guarantee that these assumptions hold in practice since many expert agents (e.g., humans) do not always strive for optimality and constraint satisfaction.
Previous works (Brown et al., 2019a;b;
Wu et al., 2019; Chen et al., 2020; Tangkaratt et al., 2020; 2021), introduced IRL approaches to learn rewards from sub-optimal demonstrations, but how to extend these methods to constraint inference is unclear.
A promising direction is to model stochastic constraints that assume that expert agents only follow the constraints with a certain probability.
Insufficient Constraint Diversity.
ICRL can potentially recover complex constraints, but our benchmark mainly considers linear constraints as the ground-truth constraints (although this information is hidden from the agent).
Despite this simplification, our benchmark is still very challenging: a ICRL agent must identify relevant features (e.g., velocity in x and y coordinates) among all input features (78 in total) and recover the exact constraint threshold (e.g., 40 m/s).
For future work, we will explore nonlinear constraints and constraints on high-dimensional input spaces (e.g., pixels).
Online versus Offline ICRL.
ICRL algorithms commonly learn an imitation policy by interacting with the environment.
The online training nevertheless contradicts with the setting of many realistic applications where only the demonstration data instead of the environment is available.
Given the recent progress in offline IRL (Jain et al., 2019; Lee et al., 2019; Kostrikov et al., 2020; Garg et al., 2021), extending ICRL to the offline training setting will be an important future direction.
D S OCIETAL I MPACT
Positive Societal Impacts The ability to discover what can be done and what cannot be done is an important function of modern AI systems, especially for systems that have frequent interactions with humans (e.g., house keeping robots and smart home systems).
As an important stepping stone towards the design of effective systems, constraint models can help develop human-friendly AI systems and facilitate their deployments in real applications.
Negative Societal Impacts Possible real-world applications of constraint models include autonomous driving systems.
Since constraint models are often represented by black-box deep models, there is no guarantee that the models are trustworthy and interpretable.
When an autonomous vehicle is involved into an accident, it is difficult to identify the cause of this accident, which might cause a loss of confidence in autonomous systems while negatively impacting society.