Learning A Risk-Aware Trajectory Planner From Demonstrations Using Logic Monitor
Anonymous Author(s)
AfﬁliationAddress email
Abstract: Risk awareness is an important factor to consider when deploying policies on robots in the real-world.
Deﬁning the right set of risk metrics can bedifﬁcult.
In this work, we use a logic monitor that keeps track of the environmental agents’ behaviors and provides a risk metric that the controlled agent canincorporate during planning.
We introduce LogicRiskNet, a learning structure thatcan be constructed from temporal logic formulas describing rules governing a safeagent’s behaviors.
The network’s parameters can be learned from demonstrationdata.
By using temporal logic, the network provides an interpretable architecture that can explain what risk metrics are important to the human.
We integrate
LogicRiskNet in an inverse optimal control (IOC) framework and show that wecan learn to generate trajectory plans that accurately mimic the expert’s risk handling behaviors solely from demonstration data.
We evaluate our method on areal-world driving dataset.
Introduction
Imagine a driver that is trying to turn across trafﬁc at an unprotected intersection where there isoncoming trafﬁc.
When faced with the situation shown in Figure 1(a), the driver needs to decidewhen it is safe to make the turn.
The driver makes this decision by evaluating how risky the nearestoncoming vehicle is in terms of the intended turning maneuver.
The risk may be dependent on anumber of factors, including how far the vehicle is from the intersection, how fast it is driving andwhether it has the intention to slow down.
If we wish to incorporate such risk-aware decision makinginto an autonomous driving system, we are faced with two problems: (1) how do we accuratelymodel the risk factors that humans take into account while driving and (2) how do we generateplans from these risk factors.
With the development of data-driven autonomous driving, we haveaccess to a growing amount of large-scale human driving data.
Large-scale data contain not onlynormal driving behaviors, but also what drivers do in order to manage risk.
Taking advantage ofthese data, our goal in this work is to develop a risk representation that can expressively describethe desired/undesired behaviors of road agents and with parameters learnable from demonstrationdata.
We aim to obtain risk metrics that reﬂect that of the human demonstrator, and generate riskaware trajectory plans using these metrics.
Recent work on risk and uncertainty aware policy learning aim to incorporate common risk constructs such as conditional value at risk, worst-case risk, etc., into the problem formulation eitheras an auxiliary loss or constraints [1, 2].
Many then use reinforcement learning (RL) to obtain thepolicy.
The main limitation of applying RL to autonomous driving is its exploration requirement.
It can be difﬁcult to explore safely especially when we wish to deal with risky scenarios.
Workon uncertainty-aware imitation learning [3] and risk-aware ofﬂine RL [4] address the explorationproblem, but are limited in the types of risks they can express.
It is important to be risk-averse in safety critical applications.
It is often more important to be averseto the right set of risks.
In this paper, we want to answer the question “if we have observed ournearby road agents for a while, can we develop a risk model that quantiﬁes how risky they are
[Figure]
based on their past behaviors in the same way that humans would.
” We take inspiration from realtime veriﬁcation [5] and formal synthesis [6] and introduce a differentiable logic monitor whichwe refer to as LogicRiskNet as an expressive and learnable risk representation.
To summarize ourcontributions,
we
• introduce a differentiable logic-based risk metric and show that this metric is able to de-
scribe complex, temporally-extended risk events as logic formulas and learn its parameters
from human driving data;
• show that we can not only use LogicRiskNet to generate risk-aware trajectories, given its
hierarchical monitoring properties, we can also use it to explain why an agent is risky;
• demonstrate in a real-world driving dataset that our method yields controllable risk-averse
or, alternatively, risk-seeking behaviors.
Note that, while we apply our approach to the domain of autonomous driving, we can equally-wellextend our approach to other domains, such as robotics.
Background: Past Time Signal Temporal Logic (PtSTL)
STL [7] offers a formalism for expressing and reasoning among a rich set of rules.
The rules aredeﬁned over predicates (inequalities of real-valued functions).
In standard STL, the formulas areforward-looking, meaning that the formulas are evaluated by looking at future trajectories.
In thiswork, we need to evaluate formulas using past trajectories (trajectories of road agents we havetracked over the past), using this past information to compute a risk of violating these rules in thefuture.
Therefore, our rules will be encoded as a set of past time STL (ptSTL) [8] formulas with thefollowing syntax
[Equation]
where a, b ∈ IR ≥ 0 , a < b , represent ﬁnite time bounds; φ is a ptSTL formula; p ( x ) < (cid:15) is apredicate; x ∈ X ⊆
IR n is a state; p ( · ) : IR n → IR is the predicate function, (cid:15) ∈ IR is a constant.¬ (not), ∧ (and) and ∨ (or) are Boolean operators.
F − (eventually) and G − (always) are past time temporal operators.
Here F − [ a,b ) φ requires φ to be true at least once in the time interval t ∈ ( − b, − a ]whereas G − [ a,b ) φ requires φ to be true for all of this interval.
Let x t 0 :
t 1 = { x t 0 , ..., x t 1 } denote the state trajectory from time t 0 to t 1 .
x t 0 :
t 1 | = φ denotes thattrajectory x t 0 :
t 1 satisﬁes φ (i.e. φ evaluates to true under x t 0 : t 1 ).
The Boolean semantics of ptSTLare provided in Appendix A.
An example ptSTL formula is F − [ τ 0 ,τ 1 ) ( | x | < 2) ∧ G − [ τ 1 ,τ 2 ) ( x > 3) which means that “ | x t | < 2 istrue for at least one t ∈ ( − τ 1 , − τ 0 ] and x t > 3 is true for all t ∈ ( − τ 2 , − τ 1 ] ”.
Similar to STL, ptSTL is equipped with a robustness degree (or robustness for short) that quantiﬁesthe level of satisfaction of a trajectory with respective to a formula.
A robustness function, denotedr , takes in a trajectory and a ptSTL formula and outputs a real-valued number.
A robustness greaterthan zero signiﬁes that the trajectory satisﬁes the given formula, i.e. r ( x t 0 ,t 1 , φ ) > 0 ⇒ x t 0 ,t 1 | = φ .
Negative robustness implies violation of the formula.
The formal deﬁnition of robustness is alsogiven in the Appendix A.
Note that each predicate relationship p ( x ) < (cid:15) appearing in φ has parameters that can be ﬁt tohow humans behave.
For example, the parameterized ptSTL formula φ =
G − [ τ 0 ,τ 1 ] ( α 1 x + γ 1 y <(cid:15) 1 ) ∧ F [ τ 2 ,τ 3 ] ( α 2 x 2 > (cid:15) 2 ) has parameters ξ = { ( α 1 , γ 1 ) , (cid:15) 1 , α 2 , (cid:15) 2 } .
Hence, we parameterize eachpredicate appearing in a formula φ , denoting the parameterized formula φ ξ (see, e.g., [9]).
Problem Formulation and Approach
Let x = ( p , ψ, v ) denote the state of a vehicle that includes position (x-y coordinates), headingand velocity (which we later use in a unicycle model).
We call the vehicle that is controlled by ourplanner the ego agent, and all other trafﬁc participants not controlled by us ado agents.
Denote Ω Tas the set of trajectories with horizon T (i.e. ω T = x t : t + T ∈ Ω T ).
Suppose that the ego vehicletracks its nearest N ado agents as well as itself via a tracking algorithm (such as a Kalman ﬁlter).
This algorithm returns a set of belief states at time step t , h t = (cid:2) x te , Σ te , x ta i , Σ ta i (cid:3) , i ∈ [0 , N ) ,which are concatenated into a history of length H as h tH =
[ h t − H , . . .
, h t ] .
Here, x te,a i and Σ te,a iare the mean values and covariances for the ego and i -th ado vehicle states.
H H is the set ofsuch histories (i.e. h tH ∈ H H ).
Deﬁne a dataset D =
[Ω T , U T , H H ] where H H is obtained fromsensors, Ω T is the ego vehicle’s ground truth future trajectory, u T = (cid:2) u t , ..., u t + T − 1 (cid:3)
∈
U T isthe corresponding sequence of controls (speed and steering).
Let feature f : Ω T
× U T
→ R bea function that maps a trajectory of states and controls to a real value (e.g. average distance to thecenter lane).
f ( ω T ) =
(cid:0) f 1 ( ω T ) , ..., f n ( ω T ) (cid:1) ∈
R n is a vector of such features.
Our problem isformally deﬁned as follows
Problem 1.
Given (1) a parametric ptSTL formula φ ξ that deﬁnes the behavior of a safe ado agent,(2) a feature f φ ξ representing ado agent risks, (3) a set of user deﬁned features representing behaviors of the ego agent f , (4) a demonstrations dataset D that contains a human driver’s decisionsand environment information, ﬁnd a trajectory distribution P θ : Ω T → [0 , 1] parametrized by θ ( θcontains the ptSTL parameters ξ as well as feature combination weights) that best matches humanbehavior; i.e., minimizes the following objective functions
[Equation]
where ω T is a trajectory from the demonstration dataset.
˜ ω T is a trajectory sampled from P θ .
Problem 1 deﬁnes a feature matching problem similar to many inverse reinforcement learning(IRL) [10] and IOC [11] formulations.
We design a set of features f to capture driving preferencesin non-risky situations, but also include a risk-based feature f φ ξ capturing the risk managementpreferences seen in demonstrations.
From the demonstration data we then learn the risk metric parameters ξ and the weights that combine the feature values.
This combination gives us a cost modelthat, when solved, yields driving behaviors similar to the demonstrator (in the context of our deﬁnedfeatures and risk models).
While existing IOC approaches are able to imitate standard driving behaviors such that the generated trajectories from the cost model are exponentially more preferred bythe agent (under maximum-entropy formulations, e.g. [11]).
In our case, the additional risk feature provides the capacity to generalize our model better under risky situations without hindering theability to replicate the driving style of human demonstrators in normal scenarios.
Logic Monitor Guided Risk-Aware LfD
In this section, we will describe in detail our risk-aware LfD framework.
We will ﬁrst introduce the
LogicRiskNet - a differentiable syntax tree generated from a parametric ptSTL formula.
The networktakes as input the tracking histories h T and outputs a risk assignment for each tracked vehicle basedon their past behaviors.
This risk assignment can be used to monitor which vehicles are risky in agiven scene.
We then show how LogicRiskNet can be incorporated into an IOC framework that isable to learn this syntax tree from demonstrations while simultaneously learn a cost function thatspeciﬁes a risk-aware planning problem, from which a trajectory can be solved efﬁciently.
Our method is depicted in Figure 1(b) where there are 2 nested loops.
The outer loop starts with themap and agent information which are passed through the LogicRiskNet to construct a risk map.
Thisrisk map constitutes the objective function for a trajectory planner.
The generated trajectory servesto control the ego agent in a receding horizon fashion.
In the inner loop, the same set optimizedtrajectories are also compared against human demonstrated trajectories in the same situations, andtheir differences serve as training signals to the LogicRiskNet.
LogicRiskNet - a differentiable parametric ptSTL risk monitor.
We describe how we may construct a ptSTL risk monitor based on a learned stochastic risk measure that can be applied to aprobabilistic description of human behaviors.
We extend the deﬁnition of robustness degree (introduced in Section 2) to encompass belief states.
In order to calculate the risk of a trajectory of beliefstates w.r.t a ptSTL formula φ , we need to modify the deﬁnition of the robustness.
Recall that thebasic element of φ is a predicate of the form p ( x ) < c , deﬁne α ( x ) =
c − p ( x ) as the predicate (thisis same as the robustness degree for the predicate).
Since x is stochastic, rather than deterministic,we can evaluate ptSTL formulas instead using a risk measure ρ : X → R , representing expectation,mean-variance, value-at-risk, etc.
In this work, we assume the expectation risk measure, but canequally well replace this with others.
We can then apply this risk measure to the robustness deﬁnitionyielding α ρ ( x ) =
ρ ( α ( x )) , x ∈ X
(This treatment is similar to [13]).
Given a parametric ptSTL formula φ ξ , a risk measure ρ ( · ) and the tracking history h H , we canconstruct the robustness risk for φ ξ as
[Equation]
where r ρ (cid:0) h H , φ ξ (cid:1) is the robustness applied to the risk-based predicate α ρ .
Once the risk predicatevalues are calculated at each time step in the trajectory, the robustness calculations for Booleanand temporal operators can be carried out.
In order to tune the parameters ξ using demonstrationdata, we follow the approach in [14].
Speciﬁcally, if we replace the max and min functions in therobustness deﬁnition with softmax approximations, we can construct a computation graph from R φwith the predicate parameters at the leaves and the robustness risk at the root.
This makes it possibleto learn ξ using backpropagation.
The intuition behind Equation 3 is that given a ptSTL formula φparameterized by ξ , agents’ past observations h H , and a risk measure ρ , the robustness risk R φ canbe calculated to represent agent risk levels in terms of satisfaction of φ .
We provide a step-by-stepexample of constructing the LogicRiskNet in Appendix F.
LfD using LogicRiskNet.
With LogicRiskNet R φ , we are able to characterize risky ado vehicles asusing the following risk feature:
[Equation]
where p te and p ta i , t ∈ [0 , T ) are the ego and ado vehicles’ positions at time t .
h iH is the trackinghistory of ado vehicle i along with the history of the ego agent.
Essentially, (4) describes a collisionfeature scaled by the risk R φ , where the inverse-squared distance may be viewed as a generalization
Algorithm
1: Inputs informa tion do 4: ω
← ConstructTrajectory
) 5: ˜ f φ ← ˜ f φ ( h H , ˜ ω
e | ξ ) , ˜ f
u ← ˜ f u ( ˜ u T ) , ˜ f N ← ˜ f N (˜ ω e , N T ) (cid:46) construct features 6:
˜ L = ˜
f φ + β u · ˜ f u + β N · ˜ f N (cid:46) construct objective 7: ˜ u T ← ˜ u T − ˜ γ ∇ u T
L
8: end
9: ˜ ω
e ←
10: return ω e , f φ , f u , ˜ f N of potential ﬁelds [15].
A safe trajectory should yield a low f φ value.
A positive R φ signiﬁes arule abiding ado agent whereas a negative R φ signiﬁes a rule-violating agent.
Intuitively, we usethe past observations to determine how risky an ado agent is, and by solving for trajectories thatminimize f φ we can control the ego agent to stay away from risky ado agents.
Our potential-ﬁeldformulation of risk acts to approximate the behavior of drivers in reacting to potential collisions butdo not over-react if collision is not imminent.
To imitate risk-free driving behavior, we additionally introduce a control effort feature and a nominaltrajectory feature for the ego vehicle as followsf
[Equation] [Equation]
) =
[Equation]
1 (cid:88) t = ||
u t || 2 where u
= ( v, ˙ ψ ) , f N ( ω T , N T ) =
T
[Equation]
(cid:88) t =
||
p te −
[Equation] [Equation]
) where p t N ∈ N T are points on the nominal trajectory.
Nominal trajectories can be the human egoagent’s trajectories (during training) or trajectories on the target lane center (during deployment).
These features encourage the ego agent to follow a trajectory (and speed) proﬁle with smooth controlefforts.
Given features f φ , f u , f N , we solve for a trajectory plan ω T by minimizing f φ + β u · f u +β N · f N using gradient descent.
Algorithm 1 describes this trajectory optimization process.
In Algorithm 1, line 4 describes the process of using a unicycle model to construct a trajectory from acontrol sequence.
This procedure allows us to generate kinematically feasible trajectories comparedto direct trajectory optimization.
In calculating the risk feature f φ , we use a constant velocity andheading model estimate of ado agents’ future trajectories p ta i , t ∈ [0 , T − 1) .
During training, thenominal trajectory N T is the ego vehicle’s ground truth future trajectory from the dataset.
Duringdeployment, this nominal trajectory is taken from points on the target center lane which can beobtained from route planning.
Algorithm 2 describes how the parameters from the LogicRiskNet along with the objective functionweights are learned from demonstrations using a maximum entropy model of uncertainty.
All entities with superscript d are taken from the demonstrations.
Line 4 calculates the sum offeatures obtained from trajectories from the human driver averaged over the dataset.
Lines 6-10calculates the same but with trajectories obtained from Algorithm 1.
Line 12 updates the parameterssuch that A f φ D and A ˜ f φ can match.
Experimental Results
We train and evaluate our method on the NuScenes dataset.
NuScenes [16] is a dataset for autonomous driving based in Boston and Singapore.
It contains 850 scenes each 20s long, containing23 object classes and HD semantic maps with 11 annotated layers.
We chose this particular datasetfor the rich semantics it provides which is well suited for rule deﬁnitions.
We will use 650 scenesfor training and
Rules used.
We use the following two rules to describe behaviors of a safe ado agent.
Our goal isto generate ego plans which avoid ado agents deemed risky in the sense that they violate these rules.
[Equation]
where H is the tracking horizon (we use H = 4 ).
φ 1 describes that a safe ado vehicle should“ always drive near the center lane and keep a safe distance from the ego vehicle.”
φ 2 expresses thatat intersections “ always if the ego vehicle is in the intersection implies that a safe ado vehicle shouldeither be far from the intersection or drive slowly.”
The ﬁnal rule φ = φ 1 ∧ φ 2 takes the form of aconjunction of a set of sub-rules.
Predicate and parameter deﬁnitions can be found in Appendix B.
Methods of evaluation.
We evaluate our method and comparison cases in terms of optimality (timeto reach goal position) and safety (minimum distance to nearby vehicles).
Within the dataset, wewill set the human ego vehicle’s start and end positions as the initial and goal positions.
Optimalityis measured as the time to travel from the initial position to within a distance to the goal positionaveraged over the validation set.
Safety is measured as the minimum distance to nearby ado vehiclesin a scene averaged over the validation set.
We perform two experiments: 1) during evaluation,we control the ego vehicle with our learned planner, and the ado vehicles move according to thetrajectories recorded in the dataset and time is synchronized; and 2) we re-run the evaluation in amore realistic setting using ado vehicles that react to the ego according to an intelligent driver model(see Appendix G).
In each of our evaluations, the ego vehicle can navigate in an environment withrealistic human or reactive ado vehicles.
We use four methods for comparison.
LogicRiskNet-IOC refers to the proposed method; Humanrefers the human driver in the dataset; BC refers to a behavior cloning agent; and TrajOpt refers toa trajectory optimization agent.
Implementation details of BC and TrajOpt agents can be found in
Appendix C.
Results and discussions.
LogicRiskNet does not only output the ﬁnal risk measure but can alsooutput the risk of all intermediate sub-formulas without needing additional computation.
This allowsus to explain the reason of its decisions.
Figure 2 shows an example of an unprotected right turnat an intersection and the vehicle that we are monitoring is highlighted by the yellow dash circle in
Figures 2(a) and 2(c).
Here we are showing two consecutive steps in the scenes.
Looking at thetop plot in Figure 2(b) we can see that this vehicle is labelled as risky (a negative risk value denotesviolation of the safety rule φ .)
To gain more insights on why this is the case, we can look at therisk values of sub-formulas.
The middle and bottom plot shows that this vehicle is neither drivingslowly nor far away from the intersection.
Figure 2(d) shows that after one time-step the risk for the for 10:
[Equation] [Equation]
[Equation]
13: end for 14: return ξ, β u , β N
[Figure]
[Table]
RiskLogicNet
[Table]
monitored vehicle has increased (risk value becomes more negative), and the cause is not becausethe vehicle has sped up but because it has driven closer to the center of the intersection.
The structureof the entire LogicRiskNet and the monitoring traces for all sub-formulas are shown in Appendix D.
Table 1 shows comparison results in terms of the time it takes to reach the goal and the minimumdistance to neighboring vehicles during this navigation.
An aggressive agent will achieve a low timeto-goal (drive faster) as well as a small safety distance.
The % Reach goal measures the percentageof scenes in the validation set the corresponding agent is able to reach the goal state.
Failure is calledif the agent does not reach the goal in the duration of the scene.
All results are averaged over scenesin the validation set (each scene is ∼ 20 seconds).
From the table we can observe that LogicRiskNetIOC achieves the most similar behavior to Human with the former slightly more aggressive.
Incomparison, TrajOpt is much more conservative.
This is because it treats all agents as the samecollision object that is needs to avoid whereas LogicRiskNet-IOC agent will place less emphasis onlow risk agents (i.e. vehicles that are stopped).
This gives the latter more free space to plan andalso behaves more human-like (as humans place different amount of risk on vehicles based on theirpast behaviors).
BC agent has a wide span of behaviors but mainly suffers from distribution shift[17] shown by its low success rate at reaching the goal.
While we use non-reactive agents in thisevaluation, in Appendix G we also extend it to a more realistic simulated environment.
Once the network parameters ξ and the objective parameters β are learned, we obtain a system thatbehaves similar to the human demonstrator.
In practice, this may not be satisfactory and the user maywish to further ﬁne tune the system.
We conduct a study to see how varying the magnitude of therisk feature in Equation (4) changes the behavior of the ego agent.
In Figure 3, the x-axis representsa risk coefﬁcient multiplied to ˜ f φ on line 6 in Algorithm 1.
We run Algorithm 1 across all scenesin the validation set and record the time-to-goal and safety distance.
Figure 3 shows that with theincreasing importance placed on the risk term, the ego vehicle becomes more conservative (drivingslower and keeping a further distance to ado vehicles).
This ﬁnding allows the user to control theego vehicle to strike a desired balance between risk-averse and risk-seeking behavior.
Related Work
Risk and uncertainty aware policy learning has been well-studied in the context of reinforcementlearning.
There have been several comprehensive surveys on this topic; [1] provide a general survey,while [2] provide greater focus on autonomous driving (AD).
In general, common risk metrics such
[Figure]
T i m e - t ogoa l
S a f e t y d
i s t an c e
[Figure]
as conditional value at risk, mean-variance, worst-case analysis, etc. have been used as either anauxiliary loss in the objective function or as a set of constraints.
In many robotic and driving tasks,however, RL is not readily applicable given its exploration requirement.
Imitation learning andofﬂine RL helps with addressing this problem.
Speciﬁcally, in [3], kernelized movement primitivesare used to estimate uncertainties in the demonstrations in ﬁnding optimal gains in a controller.
Theauthors of [4] use ofﬂine distributional RL to learn a policy that is averse to conditional value at risk.
Our work, in contrast, aims to learn from demonstrations to learn a human’s notion of risk.
It doesnot require exploration beyond the demonstrations, yet it is able to generalize well to new scenarios,and has the capacity to learn rare situations such as rule violations and near collisions.
Temporal logic (TL)-guided policy learning is an area that we take have taken inspiration from.
In this area, TL is often used to specify the ego agent’s desired high-level behavior and used togenerate rewards.
The authors of [18, 19, 20] provide surveys of recent work on the use of TL in
RL.
The exploration problem still exists in these methods.
To address these challenges, the authorsof [21, 22] learn ﬁnite state automata from demonstration and use them to guide planning with thevalue iteration network which avoids exploration.
It can sometimes be tedious to manually designa TL formula that yields satisfying behaviors.
Work has been done to make components of theformula learnable from data.
In [23], the authors propose learning linear temporal logic (LTL)formulas from demonstrations.
Given the close relationship between TL and automaton[24], theauthors of [25] propose a method that learns reward machines (an automata-like reward presentation)from demonstrations.
A shortcoming of these methods is that the LTL that they use operates onpropositions (binary variables with values true or false).
Unlike our STL-based approach, theseapproaches require discrete state and action spaces and require the demonstrations themselves tohave the same discrete representations.
Safe and risk-aware formal synthesis is an area that focuses on incorporating uncertainty and risksin planning with TL constraints.
We have taken much inspiration from [13, 12] in their techniques ofincorporating belief states into STL risk deﬁnitions.
The resulting controls are obtained by solvinga mixed-integer linear program with the STL risk as constraints.
This can be difﬁcult to scale withthe complexity of the STL risk formulas.
In [26, 27], the authors consider the uncertainties in theagent trajectories via probabilistic STL.
Methods of computing speciﬁcation-satisfying controllersunder probabilistic STL constraints are also introduced.
In these works, the probability of eventshappening at each timestep is required, which is difﬁcult to obtain and does not reﬂect how humansreason about risks in the world.
Our approach, in contrast, is data-driven and our logic risk metric iscompatible with existing model-based/model-free policy learning paradigms.
Conclusion
In this work, we introduced the LogicRiskNet - an expressive and differentiable risk representationbased on temporal logic descriptions of risky agent behaviors.
We show that with a demonstrationdataset we are able to learn risk parameters that results in trajectories similar to the demonstrator.
Wealso show that given the structure of LogicRiskNet we can monitor and explain why an agent is riskywithout additional computation.
Given our choice of the IOC framework and the interpretability ofour method, users are able to tune the objective function post-training to obtain the desired level ofrisk-averse behaviors.