T RANSCENDENTAL I DEALISM OF P LANNER : E VALUATING P ERCEPTION FROM THE P LANNING P ERSPECTIVE FOR A UTONOMOUS D RIVING
Anonymous authors
Paper under double-blind review
A BSTRACT
Evaluating the performance of perception module in autonomous driving is one of the most critical tasks in developing these complex intelligent systems.
While module-level unit test methodologies adopted from traditional computer vision tasks are viable to a certain extent, it still remains far less explored to evaluate how changes in a perception module can impact the planning of an autonomous vehicle in a consistent and holistic manner.
In this work, we propose a principled framework that provides a coherent and systematic understanding of how perception modules affect the planning of an autonomous vehicle that actually controls the vehicle.
Specifically, planning of an autonomous vehicle is formulated as an expected utility maximisation problem, where all input signals from upstream modules jointly provide a world state description, and the planner aims to find the optimal action to execute by finding the solution to maximise the expected utility determined by both the world state and the action.
We show that, under some mild conditions, the objective function can be represented as an inner product between the world state description and the utility function in a Hilbert space.
This geometric interpretation enables a novel way to analyse the impact of noise in world state estimation on the solution to the problem, and leads to a universal quantitative metric for such purpose.
The whole framework resembles the idea of transcendental idealism in the classical philosophy literature, which gives the name to our approach.
I NTRODUCTION
Autonomous driving has recently risen as a fast-advancing realm in both industry and academia, and receives a surge of interest from engineering and scientific communities (Yurtsever et al., 2020; Sun et al., 2020).
As an intricate system, an autonomous driving vehicle consists of numerous hardware components and interactive onboard modules.
As one such core component, the onboard perception module serves as the major source of real-time characterisation of the dynamic environment an autonomous vehicle (AV) navigates through.
To evaluate and improve the perception module, conventional perception tasks (such as detection, segmentation, tracking) have been well defined and corresponding performance measurements are established in computer vision to benchmark performance of perception algorithms (Lin et al., 2014).
Despite their great success in driving the development of advanced perceptual information processing modules, almost all such metrics exclusively focus on the perception-level performance in a deployment-agnostic fashion, for instance, how close a detected object is to the ground truth, while ignoring the actual impact of the result to the entire AV system.
Indeed, not all perception errors translate the same to the planning of an AV.
Obviously, miss detecting a vehicle in front of an AV is far more serious than one behind far away.
This problem is further compounded by the heterogeneity of perception errors that share little semantics in common (“How dose an error of 5m/s in velocity compare to that of a size 25% larger?”), where intuitive manual engineering is widely used (Caesar et al., 2020).
Although these issues are typically addressed by integrating road test in the real world, the process is extremely costly and time-consuming (Wachenfeld and Winner, 2016; Åsljung et al., 2017).
In result, tools are in great demand to effectively and efficiently measure the impact of perception to the whole autonomous driving system before deployment on road.
Unfortunately, these solutions still remain far less explored in the research literature.
Most recently, the community starts to approach this problem with some initial efforts (Sun et al., 2020; Philion et al., 2020; Ivanovic and Pavone, 2021;
Deng et al., 2021).
Despite some success, these preliminary solutions only exploit certain aspects of the problem, either implicitly relying on weak correlation between behaviour change and driving cost (Philion et al., 2020), inferring the holistic cost via local properties (Ivanovic and Pavone, 2021) or coarse levels (Sun et al., 2020).
In this work, we propose a principled and universal framework to quantify how noise in perception input affects the AV planning.
This is achieved by explicitly analysing the process of AV planning, in the context of expected utility maximisation (Osborne and Rubinstein, 1994) and evaluating the change of utility values critical to the AV reasoning subject to input perception errors.
Under some mild conditions (Section 3.3), we show that this planning process can be formulated as an optimisation problem with linear objective function in a Hilbert space, where utility to optimise is the inner product of an action-wise utility function and the world state distribution represented by perception.
This geometric interpretation reveals many natural and insightful properties of the problem, for example, any input error can be decomposed into two parts: one that does not affect the utility comparison ( planning-invariant error ) and the other one that directly changes the planning problem ( planning-critical error ).
Based on this novel insight, we derive a metric that quantify how a perception error changes the planning process.
We want to emphasise the necessity of understanding impacts of perception errors on an autonomous driving system via the process of planning, rather than purely from the final result (i.e., the AV behaviour, or the trajectory output from the planning module), as proposed by previous works (Philion et al., 2020).
This results from the fact that, the final planning result does not necessarily reflect how
AVs evaluate the situation, reason with the environment, and assess the costs of actions.
In fact, the correlation between behaviour change and the actual consequence is weak, or even negative in many common cases, as illustrated in Figure 1.
Actually, most works implicitly or explicitly integrate some priori knowledge of consequences of perception errors into metric design.
The complexity of such impact on autonomous driving, however, is far beyond hand-crafted rules, defeating their purposes despite tremendous amounts of manual efforts, e.g., Deng et al.
(2021) assumes that severity of an error should be weighted proportional to the reciprocal of its cubed Manhattan distance to the AV, regardless of its position relative to the AV (in front or behind the AV).
In contrast, we make little such presumption and fully rely on the planning process to infer the error consequence in a fully transparent way, which enables our solution to capture many critical cases.
In this regard, the core principle of our design resembles the idea in the philosophical system of transcendental idealism , proposed by Immanuel Kant in his classical work Critique of Pure Reason (Kant, 1998), which argues that, due to the limitation of the observer’s sensibility, cognition of external objects is processed never as they are in themselves, but via the cognitive faculties and subject to the interpretation of the observer’s experience.
For the same reason, the consequence of environment misrepresentation for an
AV due to perception errors is naturally reflected via the change in its planning (the core component of an AV that interprets its environment) and measured by the extra loss incurred, which gives the name to our framework: transcendental idealism of planner (TIP).
L ITERATURE R EVIEW
Planning for Autonomous Vehicles.
We consider the behavioural decision and motion planning as the planning process, which generates the vehicle behaviour to execute by the controller given the observation up to the planning time.
There is a rich literature to address these fundamental problems in autonomous driving, which can be roughly categorised into canonical module-based and datadriven methods.
The former relies on explicit modelling of target accomplishment in optimisation frameworks and seeks the optimal solution as the result (Schwarting et al., 2018).
The latter, on the other hand, aims to directly map raw sensor data into the AV behaviour or final vehicle control signals by leveraging the approximation power of deep learning and massive data (Bojarski et al., 2016; Grigorescu et al., 2020), which has attracted increasing attention recently.
In this work, we aim to explore the internal mechanism of a planner to gain some insights of the impacts on planning from perception noise, and specifically focus on the module-based planning with explicit target achieving process.
Extension of results in this work to data-driven planning is left for future work.
Behaviour based Metrics for Upstream Modules.
Recent works aimed to assess the performance of perception from the autonomous driving system viewpoint mostly approach the problem in heuristic ways.
Considering black-box planning models, Philion et al.
(2020) implicitly hypothesise that driving consequences of perception errors are directly correlated to the change in an AV spatiotemporal trajectories planned, and propose the planning KL-divergence (PKL) to measure the impact.
While intuitive, it fails to incorporate the context of environment and does not precisely reflect the real cost of input noise in many common traffic scenarios.
To deal with the specific problem of object representation, Deng et al.
(2021) study how object shapes can affect autonomous driving and devise the support distance error (SDE) to quantify such effect.
In a very recent work, Ivanovic and
Pavone (2021) start looking into the planning process and employs sensitivity as a probe of input signal’s contribution to AV behaviour.
This, however, implicitly leverages local-only properties of differentiable cost functions to infer global results.
In comparison, our proposed approach captures the big picture of planning process and applies to far more general cases.
AV P LANNING AS E XPECTED U TILITY M AXIMISATION
To evaluate the performance of perception module from the AV planning perspective in a principled manner, we start by introducing the preliminary basics, and then review the expected utility maximisation (EUM) as the optimal AV action framework.
After that, the interpretation of EUM in a Hilbert space is presented, based on which our metric for perception is derived.
3. P RELIMINARY
We first present the mathematical basics to facilitate the following theoretical analysis.
Unless otherwise specified explicitly, all notations follow the standardised one in Goodfellow et al.
A probability space { Φ , F , P} is defined by a sample space Φ , an event space F (a σ -algebra on Φ ), and a Borel probability measure
P on F .
A random variable X : Φ → R d ( d ∈ N ) is induced from { Φ , F , P} with distribution function F X ( x ) .
When absolutely continuous, F X ( x ) = (cid:82) x
−∞ f X ( t ) d t , where f X ( x ) is the probability density function (PDF).
L 2 ( X , ρ ) denotes the space of squareintegrable functions, and ρ is a Lebesgue measure accordingly.
A Hilbert space H =
( T , ⟨· , ·⟩ ) is defined on a complete space
T with inner-product ⟨· , ·⟩ H and induced norm ∥·∥ H .
Let S ⊂ H be a subspace of a Hilbert space
H , S ⊥ = { x ∈ H| ⟨ x, y ⟩ , ∀ y ∈ S } is the orthogonal complement of S (i.e., the set of all vectors orthogonal to S ).
The linear span of a set S is span ( S ) .
n v := v/ ∥ v ∥ is an element of unit length in a normed vector space by normalising element v .
3. A UTONOMOUS V EHICLES AS R ATIONAL A GENTS
An AV is an intelligent agent that aims to accomplish some predefined goals in an interactive and uncertain environment.
For this, an AV is constantly faced with the problem of planning in the dynamic environment, and the quality of planning determines how well the goals can be achieved.
By the classical EUM theory (Osborne and Rubinstein, 1994), at any given time t , an agent aims to achieve the maximum expected reward, defined by the utility function U , via execution of the optimal action a ∗ t such that
[Equation]
where D a,t is the set of all feasible AV actions at time t ; s ∈ S , a random variable with distribution function F
S t ( s ) , is the world state at time t in the world state space S ; and
[Equation]
Intuitively, the utility function encodes the goal or reward the AV is supposed to achieve, for example, to reach a destination in time, to minimise likelihood of collision with other objects, and to avoid sharp change in motion, and F S t ( s ) captures uncertainty about the stochastic environment given all prior world knowledge and historical observations up to t , which are estimated by modules like localisation and perception.
Architectures of many modern AV planners still follow this classical framework (Paden et al., 2016; Buehler et al., 2009; Fan et al., 2018).
3. E XPECTED U TILITY M AXIMISATION IN THE H ILBERT S PACE
To gain some insights into the expected utility of (1) and how input noise is consumed by the planning process, we introduce an interpretation in the Hilbert space to leverage geometric tools available from linear algebra.
We first establish the conditions under which a probability measure can be embedded into a Hilbert space, followed by the interpretation of EUM from a geometric perspective in Section 4.
For brevity, all proofs are left in Appendix G.
Theorem 1 (Probability Measure Embeddings in Hilbert Space) .
Let {X , d } be a compact metric space with d as the metric function, p be a Borel probability measure on
X , and X be a random variable on
X with distribution function F X ( x ) .
If F X ( x ) is absolutely continuous and the density function f
X is square-integrable, i.e., f X ∈ L 2 , then there exists a unique element 1 µ p ∈ H such that
[Equation]
where element µ p denotes the embedding of probability measure p in the Hilbert space H =
( L 2 , ⟨· , ·⟩ ) , with the inner product given by
[Equation]
The critical condition of F X ( x ) being absolutely continuous with a square-integrable density function f X in Theorem 1 is actually general and includes many popular distributions as special cases (see the discussion in Appendix F).
Theorem 1 establishes a mapping from probability measures of continuous random variables to H .
Additionally, the mapping is also injective by the following result.
Theorem 2 (Injection of Probability Measure Embeddings) .
Let p and q be two Borel probability measures defined on a compact metric space {X , d } with absolutely continuous distribution functions, then p = q almost everywhere if and only if µ p = µ q , where µ p and µ q are the embeddings of p and q in H , respectively.
A similar result for mixed distributions is also available in the appendix (Theorem 4).
Under the mild conditions in the aforementioned results, the expected utility maximisation of (1) can be rewritten as
[Equation]
Given the injective correspondence between p and µ p established above, we can leverage many tools in algebra (such as inner product, orthogonality, projection, and subspace) to analyse the impact of perception result on AV planning via the EUM in H , denoted “planning utility Hilbert space”, where the topological structure is exclusively determined by its inner product.
P ERCEPTION E VALUATION VIA AV P LANNING
In this section, we derive the extra cost of planning incurred by perception errors through the theoretical foundation established in Section 3.
While the actual world state characterisation p ( s ) consists of signals from modules other than perception (e.g., localisation), for brevity we assume that the perception module is the only source for world state estimation in the following discussion.
PCE and PIE (best moving where there is a cone in front, the belief of its position is a distribution on a line across the road p is U [ − 3 , − 2] , a uniform distribution with support [ − 3 , − 2] while the perception its location as q )
U [ − 1 , 0] .
The AV has two action options: 1) to move forward ( a ∗ , the red arrow), and the utility function is U 1 ( x )
= − 10 · 1 x ∈ [ − 1 , 1] with x being the position of the cone (only large loss for collision with the cone); 2) to make a hard brake ( a , the grey arrow), and the utility function is a constant U 2 ( x )
= − 5 (loss of hard braking is identical regardless of the cone position).
In this case, the ∆ U and ∆ µ are illustrated in the top right, while the decomposition of PIE ∆ µ ⊥ and PCE ∆ µ ∥ are on the bottom right.
Note that, ∆ µ ∥ is of the same shape
∆
(up to a negative constant)
⟨ ∆ U, ∆
⊥ ⟩
4. B REAKDOWN OF P ERCEPTION E RRORS
Consider a general case where the candidate action set is D a = { a i } , and each action is associated with a distinct utility function U ( s, a i ) ∈ H such that
[Equation]
Let a ∗ be the optimal action per EUM of (5) given the ground truth distribution of world state p ( s ) .
For a specific a ̸ = a ∗ , ∆ U ( a ∗ , a ) =
U a ∗ − U a , and the planning half-space in H is
[Equation]
Given the actual perception result q ( s ) , a ∗ is preferred over a by EUM if and only if µ q ∈ H a , i.e.,
[Equation]
with ξ ( q ; α, β ) denoting the α - β preference score given q , which exclusively decides the result of
EUM.
As illustrated in Figure 2(a), the final planning is made correctly if and only if
[Equation]
When there is an error in perception q ( s ) (i.e., (cid:13)(cid:13) µ q − µ p (cid:13)(cid:13) H > 0 ), the preference of (7) may be affected, i.e., ξ ( q ; a ∗ , a )
̸ = ξ ( p ; a ∗ , a ) , so is the preference between a ∗ and a by EUM.
To understand how the difference ∆ µ = µ q − µ p affects the result of EUM, we further decompose ∆ µ into two orthogonal components:
[Equation]
where
[Equation]
is the projection of ∆ µ onto unit vector n ∆ U (denoted behaviour direction ) parallel to U a ∗ − U a , and
[Equation]
Input
Output perception input
Get candidate perception input
Define the set D a :
D a
D a,q foreach a ∈ D a do Compute four estimates via finite-size samples:
[Equation]
where { s ( i ) q } ni =1 are n i.i.d.
observations from q ( { s t } 0 t = − τ ) , and similar for { s ( i ) p } ni =
Compute ˆ∆
; q p ) =
EU 1 −
EU 2 −
EU 3 + ˆ EUend
Compute and output the
I
) is the projection of ∆ µ onto the orthogonal complement of the subspace spanned by the behaviour direction.
In the presence of perception error ∆ µ , as illustrated in Figure 2(a), the change in preference score of (7) is only determined by ∆ µ ∥ :
[Equation]
∆ U ⟩ = ⟨ ∆ µ ∥ , ∆ U ⟩ .
For this reason, we denote ∆ µ ∥ as the planning-critical error (PCE), and ∆ µ ⊥ as the planninginvariant error (PIE).
This observation reveals a pivotal fact: not all errors in world state estimation or perception are of equivalent impact on the AV planning, and the subspace span ( { ∆ U } ) ⊥ contains all errors that would not affect EUM at all.
A toy example of PCE and PIE is shown in Figure 2(b).
Given this interpretation, we define the maximum reduction in the preference score of a ∗ over any candidate actions as the measure of impact from the perception error on AV planning:
[Equation]
4. E VALUATION OF P ERCEPTION E RROR I MPACT BY TIP
In practice, combining (7) and (12), the evaluation of perception error impact can be reduced to the calculation of four expected utilities:
[Equation]
Computing these expectations in analytical forms typically requires strong assumptions on the forms of both utility and distribution functions, which substantially limits the flexibility and representation capacity.
To allow for maximum representation flexibility, we resort to numerical methods and estimate the expected utilities from finite-size samples of world states, and show that the solution is both statistically consistent and uniformly efficient under the mild conditions in Theorem 3.
Specifically, for a fixed action a , given an i.i.d.
sample of the utilities { U ( S i , a ) } ni
=1 with S i drawn from the state distribution p S ( s ) , an unbiased estimator of the expected utility based on U-statistics is
[Equation]
Under many common practical conditions, we show that fast convergence rate via uniform bound of the estimator of (15) can be achieved by the following observation.
Theorem 3 (Exponential Convergence Rate) .
If there exists an M ∈ R such that U ( S, a ) <
M almost surely, then
[Equation]
The exponential convergence rate at O ( e − n ) provided by Theorem 3 is significant in the sense that it depends on (i) neither the dimensionality of the original state space S (i.e., the curse of dimensionality is not invoked), nor (ii) the distribution p S ( s ) and utility functions (i.e., U ( S, a ) and p S ( s ) can take any arbitrary forms).
To facilitate the understanding of our approach, the pseudo code for evaluating
TIP is provided in Algorithm 1, which sketches the basic routine to compute TIP score of a perception input sequence q ( { s t } 0 t = − τ ) from t = − τ to t = 0 for planning at t = 0 .
[Figure]
Note that, NDS saturates in several cases once the noise reaches some levels, and sensitivity of TIP to various types and levels of noise is generally more consistent than the other three (SDE-APD is computed by SDE-APD@ t =1s for velocity noise)
[Figure]
A miss detected stationary obstacle is placed in front of an AV at x =0 moving at 14m/s along the + x direction.
AV-1 (“jerk-averse”) is optimised for driving comfort with braking capped at -4m/s 2 , while AV-2 (“collisionaverse”) is for safety and can brake as much as -6m/s 2 .
Both SDE-APD and PKL consider closer miss detec- tion is worse than further ones.
TIP, however, reveals that AV-1 regards the one at 30m the worst, since the collision is inevitable even if the obstacles at 20m and 25m are detected; yet miss detection at 30m leads to a collision that could have been avoided otherwise.
In contrast, no other metrics provide this fine resolution at this level.
When the miss detection happens behind the AV, both TIP and PKL ignore its impact unlike the other two.
Note the symmetry of NDS and SED-APD in both direction of the x-axis.
E MPIRICAL S TUDY
In this section, we evaluate how the proposed TIP works in practice via extensive experiments on both synthetic and real data.
Due to the space constraint, more details are left in the appendix.
5. B ASIC S ETTINGS
All AVs used in the experiment are based on the same type of regular passenger vehicles.
The planner deployed in the autonomous driving system is derived from the popular open-sourced project of
Apollo (Fan et al., 2018).
It consists of various sub-modules of routing, object motion prediction, cost generation, path finder, and trajectory optimisation.
At each planning time instant, these submodules (except the trajectory optimisation) analyse the environment, input history, and establish the target utility function for final trajectory optimisation, i.e., the objective utility function U ( · , s ) is first created with the perception input as (part of) hyper-parameters.
The path finder then provides multiple initial paths as candidates for path-wise trajectory optimisation, and the final choice is determined by a utility decider.
The goals the planner strives to achieve include motion smoothness, traffic rule compliance, safety, progress to the destination, etc .
The planner has been extensively verified via rigorous road tests in major cities with millions of population (See Appendix B for more details).
All experiments are implemented in scenarios as the standard protocol in autonomous driving (Riedmaier et al., 2020).
Scenarios used are collected from real world road tests (see more details in Appendix D).
We consider the planning problem at a particular frame in a scenario at a time, and evaluate the utility of an action (a spatiotemporal trajectory the AV executes) for the next three seconds, following the basic setup of (Philion et al., 2020).
For comparison, three baselines are adopted from the spectrum of perception metrics: 1) at the conventional end, nuScenes dataset score (NDS) (Caesar et al., 2020) combines several traditional scoring results for 3D object detection into a single performance measure, 2) SDE average precision distance-weighted (SDE-APD)
(Deng et al., 2021) focuses more on support distance errors near the AV in an ego-centric fashion, and 3) PKL (Philion et al., 2020) serves as the representative for AV behaviour-driven metrics.
5. R ESULTS ON S YNTHETIC D ATA
In the first set of experiments, we aim to gain some understanding of various metrics in reaction to common types of perception noises.
A dataset is synthesised from our curated road test scenarios by adding controlled noise to the 3D object ground truth of vehicles, to enable clear observation on
[Figure]
Figure the sensitivity of metrics to specific types of perception errors.
For this, 1000 5s-long scenarios are assembled, with the number of objects per scene between 30 and 500, where AVs are moving on an average speed at least 5m/s.
The ground truth is annotated by professionally trained human operators.
All objects in the scenario are labeled with location, heading, category, and bounding box from 3D point clouds recorded from LiDARs on AVs during road tests.
In total, six types of errors are considered.
The false positives are tested by adding “ghost” vehicles scattered within a 70m-by-30m box centred at the AV, with motion properties randomly perturbed from the AV.
The miss detection is created by removing objects from the ground truth randomly with a certain probability (i.e., miss detection rate).
Other noises involving location, yaw, velocity and size are sampled from zero-mean Gaussian distributions with different variances and added to corresponding properties of objects in ground truth.
The comparisons are shown in Figure 3.
While all metrics are negatively correlated with all six types of perception noises, NDS saturates in some types (e.g., velocity) due to its design.
SDE-APD, also involving manual engineering, exhibits varying sensitivity at different noise levels (especially for the velocity, as the default matching threshold 0.2m is easily overwhelmed by speed noise larger than 1m/s.
Selectivity of TIP tends to be more consistent than PKL, in the sense that, while both may predict results at similar dynamic ranges when the noise is mild, the former indicates larger loss when input perception errors intensify in most cases.
We further investigate behaviour of TIP in some individual cases.
In a typical miss detection scenario, we remove a stationary vehicle in front of or behind moving AVs.
As shown in Figure 4, outcomes rendered by TIP change with different planner settings, and it predicts the miss detection of the worst loss at the border of collision events when the accdident could have been avoided by a small margin if the obstacle is successfully detected.
This reveals the superior resolution of TIP in identifying critical events from the planning perspective that would have been missed by other baselines.
The case also shows that NDS and SDE-APD fail to distinguish errors at both sides of the AV, due to their spatial or directional homogeneity by design.
5. R ESULTS ON R EAL D ATA
In the second set of experiments, we study the results from the real perception module deployed on our AVs, which is exemplified by a 3D object detection model that predicts class, location, heading, velocity and size of objects from LiDAR point clouds.
TIP is independent of the specific detector and can be applied to various methods (Lang et al., 2019;
Yin et al., 2021; Shi et al., 2020).
We adopt an end-to-end multi-view fusion (MVF) based model to synergise the birds-eye and perspective views of point clouds (Zhou et al., 2019).
The model is trained on 780K LiDAR sweeps with annotations of vehicle, pedestrian and cyclist with detection range [-67.2m, 124.8m] ×
[-51.2m, 51.2m].
A typical challenge in developing the perception model is to determine how much training is needed to reach a satisfactory level of performance.
Conventional solutions require a variety of heterogeneous
[Table]
[Table]
metrics to measure different aspects of the algorithm, including mean average precision (mAP) for detection and mean squared errors in predicted motion properties.
Recently, some unified metrics like
NDS (Caesar et al., 2020) are also proposed by manual engineering.
These metrics hardly predict outcome of the driving quality improvement of a perception model change, and in most cases the conclusion can only be made from large-scale real road tests, which is almost infeasible for such purpose (Wachenfeld and Winner, 2016; Åsljung et al., 2017).
We evaluate the performance of our 3D object detection model on the same benchmark as in Section 5.2 and compare the model output against the ground truth.
The model is trained for 15 epochs, and results are reported in the left of Figure 5.
Not surprisingly, NDS tends to increase as the model training progresses and the final checkpoint models usually achieve the best performance since
NDS combines the errors that are aligned with the loss functions optimised during training.
When evaluated with the AV involved, however, the observation is not quite similar.
SDE-APD implies that the training, without the AV context, seems to struggle with improving results on close-by objects as the losses are dominated by large number of far-away yet more challenging objects.
From either behaviour or planning perspectives, both TIP and PKL indicate that the last checkpoint model is not among the best possible models during training.
Instead, models somewhere in the middle of the training can provide better autonomous driving performance.
Actually, neither TIP nor PKL is improved significantly beyond the 7th epoch, suggesting that early termination of training may be even more beneficial to driving quality.
More importantly, we notice that TIP disagree with PKL on scenarios across models of top performance, where there are quite some critical cases identified by TIP yet missed by PKL.
The difference is illustrated in the middle of Figure 5 by the scatter plot of randomly sampled scenarios, where the PKL values are almost zero while the TIP scores are non-trivial for quite a number of cases, suggesting the drastic impact of the perception errors on the AV planning process despite similar AV behaviour outputs with or without these errors (related individual examples are discussed in Appendix C.1).
To compare other 3D detectors for offline applications (e.g., auto labelling (Qi et al., 2021)), we implement two offline models with PillarNet (Shi et al., 2022) enhanced with transformer modules as the basic detector.
The first one, denoted MVF (PN-T), uses the point cloud only from the current frame for prediction.
The second one, denoted 5F-MVF (PN-T), leverages 5 consecutive frames around the current one to predict.
Results are reported in Table 1.
Both offline models, with far less restriction on resources, have better performance by SDE-APD.
MVF (PN-T), however, cannot produce precise velocity out of observation from only one frame, which leads to inferior performance by other three metrics (MVF + tracking is the onboard model discussed above).
5F-MVF (PN-T) delivers overall best results across all metrics, despite the marginal gap computed by PKL.
To further justify the soundness of the proposed approach on the scenario level, we also implement a set of subjective evaluation similar to that in (Philion et al., 2020).
We collect 258 pairs of scenarios with actual perception noises and check weather TIP, PKL or NDS disagree on the relative severity (i.e., one believes the perception error in scenario A is worse than that in scenario B while the other one thinks alternatively).
These scenario pairs are compared and rated by 10 randomly selected human drivers to decide on which is worse from the human perspective.
The result reported in Tablesuggests that human drivers side with TIP over other three baselines.
C ONCLUSION
In this work, we propose a principled framework to evaluate perception from the perspective of planning for autonomous driving.
Our approach explicitly exploits the properties of module-based planners and effectively identifies perception noises that may cause large planning change in the context of expected utility maximisation.
Extensive experiments on both synthetic and real data confirm that our approach is capable of distinguishing perception errors that would not be separated by conventional metrics or those only exclusively focusing on AV behaviours.
E THICS S TATEMENT
Autonomous driving directly involves interaction with human beings, animals and other assets in the real world.
Any performance measure tools for autonomous driving may not identify or cover all possible failure cases of AVs that may lead to negative or even catastrophic consequences.
TIP is not an exception, despite the principle to precisely reflect the potential loss in planning by design.
It is critical that any researchers and practitioners of TIP should still implement standard and comprehensive safety protocols to ensure legitimate compliance to appropriate laws and rules, to minimise the likelihood of any potential negative impacts.
R EPRODUCIBILITY S TATEMENT
The submission includes three major technical parts: theory, implementation, and empirical study.
The main results of theoretical analysis and treatments are presented in Section 3.3 and Section 4.2.
The detailed explanation and complete proofs are presented in Appendix F and Appendix G.
The computation of TIP is provided as pseudo-code as in Algorithm 1.
For the empirical study, the planner is derived from the open-sourced Apollo with configuration fine-tuned on real road test data as depicted in Appendix B. NDS is computed via the open-sourced implementation.
SDE-APD is implemented in house according to the original paper (Deng et al., 2021).
PKL is implemented following the open-sourced project.
Appendix Table of Contents
A Additional
B Planner
C. PKL . .
C.2 Comparison with IPA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D Scenario Collection
E Breakdown of TIP
F Examples and Non-Examples of Square-Integrable Density Functions
G Proofs of Theorems in the Main Text
G.1 Notations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . G.
Embedding Probability Measures in H
G.
Injection of Probability Measure Embeddings in H
G.
4 Approximation of Expectation for Discrete/Mixed Distributions in H
G.
Uniform Convergence Rate of Expected Utility Estimators
A A DDITIONAL L ITERATURE R EVIEW
Expected Utility Maximisation .
Almost all solutions proposed so far in the artificial intelligence realm to develop intelligent agents have followed the rational-action strategy as opposed to the human-like behaviour strategy, i.e., the optimal decision made by a rational agent should optimise some achievement measurement on consequences of its decision, which is, ideally, well aligned with the predefined high-level goals (Russell and Norvig, 2009).
Among many possible formulations of the outcome measurement, the expected utility (EU) hypothesis is one of the most popular frameworks in the decision theory (von Neumann and Morgenstern, 1944).
While first introduced to characterise human behaviours in microeconomics, the idea also finds great success in many other domains, including the optimal decision for artificial intelligence systems (Osborne and Rubinstein, 1994).
Embedding Probability Measures in Hilbert Space.
Embedding probability measures into a
Hilbert space has been explored in the literature for kernel methods (Berlinet and Thomas-Agnan, 2011; Smola et al., 2007).
These methodologies exclusively focus on the reproducing kernel Hilbert space, which is spanned by Mercer kernel functions.
However, the implicit requirement on function continuity may restrict its applicability to our problem, where either utility and probability density functions may be discontinuous.
In this work, we consider a less restrictive Hilbert space where only square-integrability is needed and more flexible functions are possible.
B A UTONOMOUS V EHICLE P LANNER
We start by introducing the autonomous vehicle planner, which provides the fundamental toolkit for the proposed evaluation framework.
Our planner is designed to control an Level 4 AV running in urban areas of major modern cities.
Its modulized architecture is similar to many popular utility-based planners (e.g., (Fan et al., 2018)), which consists of four major components as illustrated in Figure 6:
• The predictor infers the motion information s m in the future (i.e., t > 0 ) for all dynamic road objects from perception input history (i.e., t ⩽ 0 ) up to the planning time (i.e., t = 0 ).
• The action proposer analyses the current environment at the planning time from (1) the perception input, (2) future object motion input, and (3) other inputs (e.g., localisation, traffic lights, semantic maps, routing path, etc.), and proposes various sets of behaviours (e.g., “go straight” and “lane change”) for the AV with an initial feasible spatiotemporal trajectory for each set.
• The trajectory optimiser takes results of above components as input, and finds the optimal spatiotemporal trajectory for each behaviour set by numerically solving an optimisation problem with the initial feasible trajectory from the proposer as the starting point.
• The optimal trajectories from all behaviour sets are then submitted to the action decider , which assemblies all information to evaluate the utilities of different candidate actions (with corresponding optimal spatiotemporal trajectories), and makes the final decision on the a ∗ .
Similar to many popular architectures (e.g., (Fan et al., 2018)), our planner utility function U ( a, s ) is of the general form [Equation] where λ i are the (static) coefficients, the atomic element function
U i depending on both a and s characterises the “compatibility” of action a and scenario s , U s ( s ) depicts the current environment, and U a ( a ) evaluates the quality of the action.
These terms can be categorized into the following groups.
• The smooth motion group encourages motion without abrupt change in acceleration, and penalises large jerks (i.e., the derivative of acceleration).
• The safety distance to road obstacles group is designed to keep the AV away from other road objects to minimise the collision likelihoods.
This distance is defined as ℓ 2 distance between the AV spatiotemporal sweeping contour and a foreign object on the road.
• The legal motion satisfaction group is designed to enforce the AV to strictly follow all applicable traffic rules when in motion.
For instance, the cost for crossing solid yellow lines is so significant as that such behaviour is prohibited unless a collision cannot be avoided otherwise.
Some other legal options also come at certain prices too to discourage high-risk behaviour (e.g., lane changes in crowded scenes).
• The progress to the destination group aims to guide the AV to achieve the goal in the big picture and reach the final destination.
The aforementioned planner deployed onboard our AVs have gone through rigorous road test in urban areas of major cities with millions of population.
Results of more than 10,000 miles on average tested every week indicate that the planner achieves 111.3 miles per intervention (MPI), which suggests that the planner used in this work is a reasonable and validated one.
C M ORE C OMPARISONS TO R ELATED M ETRICS
In comparison to the other advanced metrics (PKL (Philion et al., 2020), IPA (Ivanovic and Pavone, 2021), SDE (Deng et al., 2021)) that are recently proposed for more effective perception evaluation, our approach provides a universal and principled solution to evaluate the impact of perception noise from the perspective of the planning process of an AV.
C. C OMPARISON WITH PKL
Here we provide more empirical results to better understand the difference between the proposed TIP and PKL (planning KL-divergence)
(Philion et al., 2020)
. Results on Synthetic Data
Figure 7 demonstrates a scatter plot for scene-wise TIP and PKL results on the synthetic data generated as described in Section 5.2 with 6 false positives per scene.
It is observed that some results are very close to either x - or y -axis (the top right corner), suggesting that TIP and PKL deviate in determining if a perception error (i.e., false positive) is crucial to planning on these cases.
A typical scenario of such disagreement is shown in Figure 8, where the behaviour of the AV does not change significantly with ground-truth or noisy perception inputs (PKL = -0.248), yet the planning process has changed quite a lot (TIP = -61.654)
due to the affinity of false positive objects that has drastically change the planning cost to close objects.
In this case, TIP is capable of detecting serious perception errors that PKL fails to identify
. Results on Real Data
On the real data
[Table] we also have similar observations [Table] and demonstrate the actual scene for one such scenario in Figure 9.
As shown in this case, a false detection of a vehicle in front of the AV does not change the behaviour considerably (PKL = -0.802), while the significant planning cost change is reflected by TIP with value -115.42.
More individual examples are shown in Figure
Overall, on both synthetic and real data, we show that the proposed TIP can efficiently and effectively capture perception errors critical to the planning that may be missed by PKL.
This confirms our motivation to exploit the actual AV planning process, as opposed to the AV behaviour (i
[Table]
e., the result of planning), to gain insights into the impact of input perception error on the whole AV system
[Table]
/NOdnMIfyB8/kD32SNig==</latexit> a ⇤ <latexit sha1_base64="wral/ex3pFVY4hRazmHsD2Y1gIY=">AAAB9XicbVDLSgNBEOyNrxhfUY9eBoPgKexKUI9BPXiMYB6QrKF3MpsMmX0wM6uEJf/hxYMiXv0Xb/6Ns8keNLFgoKjqpmvKiwVX2ra/rcLK6tr6RnGztLW9s7tX3j9oqSiRlDVpJCLZ8VAxwUPW1FwL1oklw8ATrO2NrzO//cik4lF4rycxcwMchtznFLWRHtJegHpEUZCbaR/75YpdtWcgy8TJSQVyNPrlr94goknAQk0FKtV17Fi7KUrNqWDTUi9RLEY6xiHrGhpiwJSbzlJPyYlRBsSPpHmhJjP190aKgVKTwDOTWUi16GXif1430f6lm/IwTjQL6fyQnwiiI5JVQAZcMqrFxBCkkpushI5QItWmqJIpwVn88jJpnVWd82rtrlapX+V1FOEIjuEUHLiAOtxCA5pAQcIzvMKb9WS9WO/Wx3y0YOU7h/AH1ucPSG2SYQ [Equation]
[Figure]
[Figure]
IPA (injecting planning-awareness) is recently developed in (Ivanovic and Pavone, 2021) to encode the planning error based on the hypothesis that the impact of object location error is proportional to the gradient magnitude of the planning cost functions involving the AV-object distance.
This solution however requires differentiability of the planning cost functions, while our approach does not and is thus more applicable to the Level 4 AVs that are typically structured as a modularized pipeline of individual components including perception, prediction, planning, etc.
Even more serious is that it fails to account for all cases since the local properties (gradients) do not always reflect the global ones (overall losses).
To illustrate this, consider a scenario, where the cost of AV being close to an object is 1 /d .
Now assume that there are the following two cases of object location errors.
• Case one: The ground truth distance of an object to the AV is 1m, and the noisy distance estimated by perception is 0.9m. Per the metric IPA defined in (Ivanovic and Pavone, 2021), the result is [Equation]
[Equation]
while the actual cost difference is (cid:12)(cid:12)
[Equation]
[Equation]
11 (cid:12)(cid:12) = 1 / 0 .
9 − 1 = 0 .
• Case two: The ground truth distance of an object to the AV is 2m, and the noisy distance estimated by perception is 2.5m. Per the metric IPA defined in (Ivanovic and Pavone, 2021), the result is
[Equation]
while the actual cost difference is (cid:12)(cid:12) 12 − 12 .
5 (cid:12)(cid:12) = 0 . 5 − 1 / 2 . 5 = 0 . 1 .
Obviously, the metric IPA of case two is larger than case one, while the actual error in planning cost is the other way, as Taylor series up to first order terms adopted by IPA cannot precisely delineate the cost function value change over large range input variation.
D S CENARIO C OLLECTION
The scenarios used in this work are curated from AV road test in real world from public roads in urban areas of megacities, e.g., central business districts, populated residential communities, major commercial areas etc.
Each scenario is a 10s-long excerpt extracted from a continuous interval of road test, which consists of 1) all raw data recordings (LiDAR point clouds, camera images, positioning signals etc.)
from the road test during the interval and 2) the portion of offline generated high-definition (HD) and birds-eye view (BEV) raster maps that cover the field of perception during the interval.
The duration of road test ranges from tens of minutes to several hours and covers various times of both weekdays and weekends from early morning till late night during a period of more than one year, providing a rich blending in weather condition (e.g., sunny, cloudy, rainy, and snowy), traffic intensity (e.g., rush hours on highways and crowded streets during holidays), road participant diversity (e.g., private cars, cyclists, pedestrians, and emergency vehicles), and so forth.
The scenarios are selected from non-trivial situations (i.e., those with few traffic participants are filtered out) with balance in AV motion speed, diversity of traffic participants, weather, geographical locations, etc.
E B REAKDOWN OF TIP
In order to facilitate understanding of the TIP evaluation process we have illustrated the evaluation steps in Figure where a typical false negative case is used for the analysis
F E XAMPLES AND N ON -E XAMPLES OF S QUARE -I NTEGRABLE D ENSITY F UNCTIONS
Theorem 1 in the main text requires square-integrability of a density function,which includes many popular cases that may be used for constructing the utility function for planning.
Example 1 (Bounded PDFs) .
If both the support and range of the PDF f ( x ) of an random variable is bounded, then f ( x ) is square-integrable, e.g., uniform distributions.
Example 2 (Parametric PDFs) .
PDFs of many popular parametric statistical models are squareintegrable, e.g., (sub-)Gaussian, (sub-)Laplace, Gamma (including exponential, Erlang, and χdistributions), etc.
Example 3 (Mixture Models of Countable Components with Square-Integrable PDFs) .
The PDF of a mixture model is of the form:
[Equation]
(17) where f i ( x ) is the PDF of the i -th component out of the countable set { f i ( x ) } .
f ( x ) of (17) is square-integrable if ∀ i, f
i ∈
L 2 and M = sup i ∥
f i ∥
H < + ∞ as
[Equation]
A variety of mixture models are included such as Gaussian mixture models and mixtures of Gamma distributions.
On the other hand, since ℓ 1 and ℓ 2 norms are not necessarily equivalent in infinite-dimensional spaces there are indeed some density functions f
( x ) ∈ L 1 with infinite ℓ 2 norm.
Non-Example 1 (Square-Unintegrable PDFs) .
Let the distribution F X of a random variable
X be
[Equation]
where a > 0 is the parameter; and the density function is then
[Equation]
where f ( x ) is not square
G P ROOFS OF T HEOREMS IN THE M AIN T EXT G. N OTATIONS
Besides the notations in Section 3.1, a few more are introduced as follows.
A unit step function is
W ( x − c ) = 1 x ∈ [ c, + ∞ ) , c ∈ R .
L 1 ( X , ρ ) denotes the space of absolutely integrable functions.
G. E MBEDDING P ROBABILITY M EASURES IN H
P ROOF (Theorem 1).
Since F X ( x ) is absolutely continuous, there exists a density function f X ( x ) ∈
L 1 such that
[Equation]
[Equation]
almost everywhere.
Since f X ( x ) ∈ L 2 , let M = ∥
f X ∥ < + ∞ , ∀ g ∈ H , we have [Equation] [Equation] |
[Equation]
(23) where (23) follows from the Cauchy-Schwarz inequality (Rudin, 1976, Theorem 11.35).
Thus, the linear functional E X [ · ] is bounded on H and
[Equation]
where µ p := f X ∈ H is the embedding of the probability measure in H .
Now assume that there exists another element µ ′ ∈ H such that E
[Equation]
Since µ p − µ ′ ∈ H , we have
[Equation]
Therefore, the embedding µ p for probability measure p in H is a unique equivalence class of the functions that are equal almost everywhere.
G. I NJECTION OF P ROBABILITY M EASURE E MBEDDINGS IN H
To prove the injection of probability measure embedding in Theorem 2, a preliminary result of (Dudley, 2002, Lemma 9.3.2) is first introduced.
Lemma 1.
If ( X , d ) is a metric space, p and q are two probability measures on X , then E x ∼ p ( x )
[ g ] =
E x ∼ q ( x ) [ g ] , ∀ g ∈ C b ( X ) if and only if p = q , where C b ( X ) is the space of all bounded continuous functions on
X .
P ROOF (Theorem 2).
Now we prove this theorem in the following two directions.
Necessity .
Since the embedding of a probability measure is unique in H , it is easy to see that µ p = µ q if p = q .
Sufficiency .
Note that, by Weierstrass extreme value theorem (Rudin, 1976, Theorem 4.16), any real continuous function g ∈ C ( X ) on the compact space X is bounded, i.e., ∀ g ∈ C ( X ) , ∃ M ∈ R such that (cid:12)(cid:12) g
( x ) (cid:12)(cid:12) <
M, ∀ x ∈ X .
It follows that C ( X ) ⊂
L 2 ( X ) since
[Equation]
Now if µ p = µ q almost everywhere, we have
[Equation]
Thus p = q by Lemma 1.
G. A PPROXIMATION OF E XPECTATION FOR D ISCRETE /M IXED D ISTRIBUTIONS IN H
While Theorem 1 in the main text only addresses the continuous distributions, a similar result can be found given point-wise continuity conditions for general distributions, which can be decomposed into absolutely continuous and discrete parts (Chung, 2000).
Theorem 4 (Approximation of Mixed Distribution) .
Let F ac ( x ) be an absolutely continuous distribution function with density function
f X ( x ) ; F d ( x ) = (cid:80) i b i W ( x − a i ) a discrete distribution function of point mass at a countable set { a i } such that b i > 0 and (cid:80) i b i = 1 ; F X ( x ) =
λF ac ( x )+(1 − λ )
F d ( x ) a mixed distribution function with λ ∈ (0 , 1) as the convex combination coefficient.
If f X ( x ) is square-integrable, and g ( x
) ∈
L 2 is uniformly continuous at { a i } , then there exists a sequence of { µ p,n } ⊂ H such that
[Equation]
We start by considering a simple discrete case by the following lemma.
Lemma 2.
Let F X ( x ) =
W ( x − a ) be a discrete distribution function with point mass at a ∈ X .
If g ( x ) ∈
L 2 is continuous at a , then there exists a sequence of { µ p,n } ⊂ H such that
[Equation]
P ROOF (Lemma 2).
∀ ε > 0 , since g ( x ) is continuous at a , there exists a a radius r > 0 such that
[Equation]
with a positive measure V = ρ ( B ( a, r )) > 0 , where B ( a, r ) ⊂
X is a neighbourhood of r around a .
Define
[Equation]
We have
[Equation]
Thus,
[Equation]
Lemma 2 implies that the expected value of a function continuous at the point mass of a delta distribution can be approximated by an inner product in H with any arbitrary precision .
P ROOF (Theorem 3).
Note that
[Equation]
Since F ac ( x ) is absolutely continuous, by Theorem 1, there exists a µ ∈ H such that
[Equation]
[Equation]
On the other hand, ∀ ε > 0 , since g ( x ) is uniformly continuous at { a i } , there exists a radius r >such that ∀ i ,
[Equation]
and V = ρ ( B ( a i , r )) > 0 (translation invariance of Lebesgue measures in R d ).
Define
[Equation]
We have
[Equation]
Thus
[Equation]
Combining (28) and (29) leads to
[Equation]
G. U NIFORM C ONVERGENCE R ATE OF E XPECTED U TILITY E STIMATORS
P ROOF (Theorem 3).
Assume that { X i } ni =1 and independent and X i ∈
[ a i , b i ] almost surely.
Let ¯
X = 1 n (cid:80) i
X i .
Per Hoeffding’s inequality (Hoeffding, 1963, Theorem 2), for any ε > 0 ,
[Equation]
By symmetry, it also holds true that, for any ε > 0 ,
[Equation]
Combining one-side inequalities of (30) and (31) leads to
[Equation]
where M = sup( {| a 1 | , · · · , | a n | [Equation] | b 1 | [Equation]
[Equation] On the other hand, Bernstein inequality (Bernstein, 1946) also provides an improved revision of
Chebyshev’s inequality by incorporating both almost-sure bound and variance bound:
[Equation]
The proof is completed by setting X i
=
U ( S i , a ) and taking the lowest bound of (32) and (33) for the tail probability of (cid:12)(cid:12)(cid:12) ¯
X − E (cid:2) ¯ X (cid:3)(cid:12)(cid:12)(cid:12) .