Just Label What You Need: Fine-Grained Active Selection for P&P through Partially Labeled Scenes
Abstract: Self-driving vehicles must perceive and predict the future positions of nearby actors to avoid collisions and drive safely.
A deep learning module is often responsible for this task, requiring large-scale, high-quality training datasets.
Due to high labeling costs, active learning approaches are an appealing solution to maximizing model performance for a given labeling budget.
However, despite its appeal, there has been little scientiﬁc analysis of active learning approaches for the perception and prediction (P&P) problem.
In this work, we study active learning techniques for P&P and ﬁnd that the traditional active learning formulation is ill-suited.
We thus introduce generalizations that ensure that our approach is both cost-aware and allows for ﬁne-grained selection of examples through partially labeled scenes.
Extensive experiments on a real-world dataset suggest signiﬁcant improvements across perception, prediction, and downstream planning tasks.
Introduction
For self-driving vehicles to safely plan a route they must perceive nearby actors and predict their future locations.
In a self-driving stack, a learned perception and prediction (P&P) model is responsible for this task, taking raw sensor data as input and producing object detections and future predictions.
These models require large-scale, high-quality training datasets due to the high dimensional sensor inputs and long tail of future outcomes.
While self-driving companies collect massive amounts of data from real-world driving, annotating the data remains a major bottleneck.
Furthermore, some of the data may be less interesting for model training – e g., a prediction dataset with many parked vehicles is less informative than one with highly interactive, moving actors.
Therefore, the choice of which examples to label is crucial to maximize performance for a given budget.
Given a particular model, we seek to determine the examples most likely to improve performance when labeled.
This problem is well-studied in the ﬁeld of active learning and recent work has shown impressive performance gains over random selection in many tasks, including image classiﬁcation, semantic segmentation and 2D object detection [1, 2, 3].
Active learning presents a promising framework to employ in real-world self-driving development, where models can continually improve as new batches of examples are selected iteratively for labeling (see Figure 1).
Despite the appeal of active learning, few approaches have been developed for self-driving.
Scientiﬁc analysis is limited to object detection [4, 5], with no approaches designed for P&P.
When applying active selection to this task, we ﬁnd the traditional formulation to be ill-suited.
First, while approaches typically assume ﬁxed labeling costs per example, annotation costs can vary drastically as they depend on the number of actors present.
Furthermore, the spatial label structure can be exploited to support partial labeling, enabling ﬁne-grained active selection.
Speciﬁcally, with small modiﬁcations, P&P models can be trained from partial supervision.
This allows the active learner to select speciﬁc actors in a scene without requiring the remaining actors to be selected, as they may be
[Figure]
Params: K , N
Inputs: X U , X (0) L , M (model)
Outputs: X ( N ) L , M 1: for i ∈ 1 . . .
N do 2:
S ( i ) ← score( M , X U \ X
( i − 1) L ) 3:
Q ( i ) ← select top k
( S ( i ) , K ) 4:
X ( i +1)
L ← X ( i ) L ∪ Q ( i ) 5:
M ← train ( M , X
( i +1) L ) 6: end for uninteresting for model improvement.
With these differences in mind, we introduce a ﬁne-grained, cost-aware selection along with speciﬁc scoring criteria for active selection in the P&P setting.
We leverage a real-world, large-scale dataset to analyze the effects of partial labeling and active selection for P&P.
First, we study models trained on partial supervision, without active selection, and observe improved performance from sparser labeling.
Next, we analyze the added beneﬁts of ﬁne-grained active selection and observe signiﬁcant gains compared to traditional approaches.
Further analysis shows the gains are most signiﬁant on rare events, translating to improvements in downstream planning performance.
All together, our analysis suggests that the dominant paradigm of labeling entire self-driving scenes is not the most efﬁcient use of a ﬁxed labeling budget and that more ﬁne-grained active selection may be required to most effectively select examples for labeling.
Related Work
Perception and Prediction: While perception and prediction have traditionally been handled separately, [6, 7] introduce models to jointly perform both tasks, improving performance and efﬁciency.
Among the exciting progress made in both tasks over recent years, most relevant to our work are improvements in prediction representations, allowing models to better characterize uncertainty.
Examples of representations include trajectories [7, 8], probabilistic occupancy maps [9, 10], Gaussian mixtures [11], implicit latent variable models [12], and auto-regressive models [13].
Active Learning: We focus on pool-based active learning, in which new training examples are queried from a large, unlabeled pool [14].
One class of approaches seeks to characterize model uncertainty, measured via model disagreement [15, 16], entropy [17, 18], a learned loss prediction [1], or a discriminator score [19], and select examples with high uncertainty for labeling.
While often effective, uncertainty based approaches are prone to selecting a subset of similar examples when computational constraints require large batches of examples to be selected before retraining.
This motivates diversity-based approaches [20, 21, 22, 23] which seek to ﬁnd a representative subset of the unlabeled pool.
[3, 24, 25] introduce approaches which balance both uncertainty and diversity in the selection process.
Most related to our domain are [4, 5, 26, 27, 28] which study active learning approaches for object detection.
While most approaches assume ﬁxed labeling costs per example, [29, 30] have explored explicitly modeling individual labeling costs as part of the selection process.
Leveraging partially labeled data for ﬁne-grained active selection has been explored in semantic segmentation [31, 32] and more generally, in the context of structured prediction problems [33].
Dataset Selection: Many self-driving datasets select examples manually [34], randomly, or via hardcoded rules.
Argoverse describes rules-based criteria to mine interesting trajectories for prediction [35].
More recently, [36] proposed a set of complexity measures for dataset selection.
Finally, [37] proposed tagging attributes of self-driving scenes, enabling retrieval for dataset curation.
Active Learning for P&P
Given the high labeling costs of P&P datasets, budgets can be spent more efﬁciently by intelligently selecting examples for labeling.
Active learning offers a promising solution, selecting examples believed most likely to improve model performance.
In this section, we review traditional poolbased active learning, where examples are iteratively selected from an unlabeled pool to build a high-quality labeled dataset.
Then, we address shortcomings in the P&P setting by introducing a new paradigm which is both cost-aware and enables ﬁne-grained selection through partially labeled scenes, providing the ﬂexibility to ensure budgets are spent effectively.
Finally, we provide concrete selection criteria used within our framework to optimize the model’s performance.
3.1 Traditional Active Learning
Self-driving companies can collect large-scale unlabeled real-world data when operating their vehicles.
Our goal is to select the best subset to label to improve model performance.
We assume access to a large, unlabeled pool of examples, X U , and an initial subset of labeled examples, X (0) L .
Each example x ∈ X U represents an input to our model f ( x ) and if selected, a labeling oracle returns the ground truth supervision, y =
L ( x ) .
In the P&P setting, inputs x represent raw sensor observations and HD Maps, and labels y represent actor bounding boxes at the current timestep and for the prediction horizon of T seconds.
In each active learning iteration, we select a subset from the remaining unlabeled examples, Q ( i ) ⊂
X U \ X ( i − 1) L , query the labeling oracle, and add the examples to our labeled set.
With each iteration, the model is retrained or ﬁne-tuned with the latest dataset.
Traditionally, the active learner will select a ﬁxed number of examples at each iteration, | Q ( i ) | = K .
This implicitly assumes that each example x ∈ X U can be labeled for the same cost, an assumption clearly violated in the P&P setting, which we will relax in the next section.
While a variety of approaches have been studied for active selection, we focus on methods which produce a scalar score for each example, S ( x ) ∈ R .
Scores represent some notion of informativeness where highly scored examples are believed to be most likely to improve model performance.
For example, measures of model uncertainty, such as entropy, are commonly used (see Section 3.3 for concrete scoring functions for P&P).
As different models may beneﬁt from different types of examples, most scoring approaches depend on the model’s current state.
After scores have been computed for the remaining unlabeled examples, the top K examples can be selected for labeling.
This process repeats for N active learning iterations and is summarized in Algorithm 1.
3.2 Fine-Grained Cost-Aware Active Learning
In this section, we generalize two aspects critical to the P&P setting, allowing for variable labeling costs and ﬁne-grained selection through partial supervision.
Cost-Aware Active Learning: As a self-driving vehicle operates, the surrounding environment will change, leading to scenes with different labeling costs.
Crowded scenes can contain hundreds of actors, which are each traditionally labeled with a precise bounding box.
Sparser scenes, on the other hand, can be labeled with little manual effort.
To account for these differences, we explicitly model the cost to label each example, C ( x ) .
At each iteration, rather than select a ﬁxed K examples, the learner is instead given a ﬁxed budget B , which cannot be exceeded, (cid:80) x ∈ Q ( i ) C
( x ) ≤ B .
This formulation is a generalization of the previous setting, which can be recovered by setting
C ( x ) = 1 for all examples and B = K .
In practice, labeling cost for P&P examples can be accurately modeled as a linear function of the number of actors in the scene as most annotation time is spent drawing detailed bounding boxes for each actor.
This new formulation requires modiﬁcations to our selection algorithm, since high scoring examples S ( x ) may also have high costs C ( x ) .
Therefore, rather than sorting by score, we can select examples with the highest value, V ( x ) =
S ( x ) C ( x ) .
Since the cost of labeling a region is unknown before labeling, we approximate it using the number of detections after non-maximum suppression (NMS) [38].
After labeling, the true cost is known, and active selection can continue iteratively until the budget is reached.
Since examples in the P&P setting represent large scenes, scores S ( x ) and costs C ( x ) can vary signiﬁcantly as scenes can have few to many actors, each contributing to the total score and cost.
As a consequence, coarse-grained scoring will be suboptimal as scenes may contain regions with high score and low cost (e.g., a single car performing a rare U-Turn) and other regions with low score and high cost (e.g., a parking lot ﬁlled with many static vehicles).
This motivates the need for more ﬁne-grained scoring and selection.
Next, we describe modiﬁcations to support partially labeled scenes, which will enable ﬁne-grained selection for better performance in the cost-aware active learning setting.
Partially Labeled Scenes: We generalize the labeling process to allow for partial labeling.
Along with the added ﬂexibility, this setting is also realistic in practice.
Even as entire scenes are labeled today, annotation platforms often decompose work into subtasks, which can be more easily distributed and validated across a labeling team.
As a simple extension, platforms could support querying labels for only particular regions.
To support partial labels, we redeﬁne an example x R as the scene augmented with a labeling region R , x R = ( x , R ) .
Given the set of labels for the entire scene y
=
L ( x ) and a region R , each actor’s bounding box label y i will either be fully contained in R , completely outside of R , or partially inside of R .
For simplicity, we assume that if any part of the bounding box y i of an actor is inside R then it will be provided as a label.
In practice, this translates to labelers annotating all actors, even those that are only partially visible in the labeling region.
More formally, the labeling oracle returns labels for an example x R , L ( x R ) =
{ y i :
y i ∈
R and y i
∈
L ( x ) } .
Training from Partial Supervision: We adapt training to support partial supervision by applying the loss only on the labeled region, R .
Importantly, we do not alter the network input x , since we do not want to bias the network by changing input statistics.
Therefore, the forward pass remains unchanged, ˆ y = f
( x ) .
To compute the loss, we only consider the labels that we have received in R ,
[Equation]
Here, (cid:96) P ( · , · ) represents traditional multi-task perception and prediction losses applied over the positive examples in R and (cid:96) B ( R ) represents a “background” loss which encourages the network not to output detections for negative regions in R .
For example, in our experiments, (cid:96) P includes a probabilistic prediction loss, a bounding box regression loss and cross-entropy on positive examples, whereas (cid:96) B ( R ) represents the hard negative mining loss, sampling only negative anchors from R .
Fine-Grained Selection:
Without restrictions on R , there are inﬁnite regions to consider for a given scene.
Therefore, in order to efﬁciently score and select regions for labeling, we consider the set obtained by discretizing the entire scene into a rectangular grid.
Speciﬁcally, we divide each example x into HW non-overlapping regions, x R = ( x , R h,w ) .
By setting H =
W
= 1 , we obtain a single region for each scene and recover the original formulation.
As H and W increase, candidate regions become smaller, providing the learner more ﬁne-grained precision for selection.
Most steps of the selection process can remain unchanged.
Scoring functions now operate over examples augmented with regions, S ( x R ) returning a score that only considers network predictions in R .
Similarly, only the cost of labeling the queried region C ( x R ) is incurred when selecting x R .
As regions sizes shrink, we observe that the active learner is more likely to select a large number of scenes, each labeled with very sparse supervision.
While this dataset would contain many interesting actors, we also ﬁnd this unconstrained selection results in signiﬁcantly longer training times as examples are less densely labeled.
Additionally, we observe training instabilities due to the imbalances between the amount of supervision available for each example.
To alleviate these issues, we introduce a sparsity regularizer, which requires that the active learner select a minimum number of positive examples M for any selected scene.
Therefore, letting P ( x R ) represent the number of positive examples in an example, our new formulation can be summarized by the following optimization problem solved by the learner at each iteration,
[Equation]
We solve this optimization greedily by ﬁrst selecting the highest scoring scene remaining, then selecting the highest regions in the scene until at least M actors are labeled.
We continue selecting new scenes until the budget is reached.
[Figure]
t
[Figure]
r
[Figure]
r =
[Figure]
r = 1/ r = 1/
[Figure]
3.3 Selection Criteria
Our ﬁne-grained, cost-aware active learning formulation introduced above generally supports any approach which provides an informativeness score S ( x R ) per example.
In this work, we focus on uncertainty-based approaches, an extremely common active learning paradigm based on the assumption that training on uncertain examples are most likely to improve future performance.
As this approach depends on a model’s characterization of uncertainty, we ﬁrst describe our probabilistic
P&P model, followed by possible uncertainty-measures that can be used for scoring.
Model: Following [6], we jointly train a model for both perception and prediction from LiDAR and HD map inputs.
A model which naturally characterizes uncertainty over predictions is desirable, as these uncertainty estimates provide a useful measure of informativeness for scoring.
Therefore, we leverage the output representation of [11], using a mixture of K Gaussians to represent the distribution of each actor’s future positions.
For simplicity, independence is assumed between timesteps of the prediction horizon.
Thus the likelihood of a particular actor trajectory, y i , can be written as,
[Equation]
where N is the pdf of a 2D multivariate Gaussian with parameters µ tk , Σ tk , and π k represent Gaussian mixture weights.
These parameters, for each detected actor, are predicted by a deep neural network trained with negative log likelihood (see Section 4 for more details).
With knowledge of the model, we now introduce measures of uncertainty to use as selection criteria.
Due to the multitask nature of the task, we present separate selection criteria for the detection and prediction task.
In practice, a mix of both can be used to ensure performance improves across both tasks.
Detection Entropy: We focus on characterizing the uncertainty over the model’s classiﬁcation predictions for each anchor.
For classiﬁcation tasks, the uncertainty is typically estimated by calculating the entropy of the model’s predicted probabilities.
Given anchors a ∈ A with associated probabilities p a , the entropy of the predictions are given by,
[Equation]
To score regions, we assume independence between anchors and sum the entropies of anchors in R .
Prediction Entropy: Computing prediction entropy naturally depends on the model’s output representation.
Our model outputs a Gaussian mixture for each predicted actor.
Unfortunately, there is no known closed form solution to computing this distribution’s entropy [39].
Therefore, we are required to estimate the entropy via approximations.
We explored various approximations, including a sample-based monte-carlo estimate, but all performed similarly to or worse than an approximation via the entropy of the discrete categorical distribution induced by the mixture weights π k ,
[Equation]
Intuitively, this approximation is well-suited to capture cases where the model is uncertain between multiple possible modes, which is likely representative of the true entropy of distribution.
We use
[Table]
this approximation due to its simplicity and computational efﬁciency while providing similar performance.
Finally, we assume independence and sum the actor entropies within each region.
Experiments
In this section [Table] we analyze the effects of partial labeling and ﬁne-grained active selection.
First, we explore partial labeling independent of active selection.
Next, we explore the improvements provided by active selection for prediction.
We ﬁnd that a simple prediction entropy combined with ﬁne-grained active selection outperforms various traditional scene-based approaches.
More detailed analysis shows that ﬁne-grained selection enables the learner to better oversample labels exhibiting complex driving behaviors, resulting in better performance on these challenging behaviors in the test-set.
In practice, we are most interested in the effects of these improvements on the downstream motion planning task, where we ﬁnd signiﬁcant improvements across most metrics.
Finally, we observe similar improvements for perception when using detection entropy as the selection criterion.
Dataset: We leverage a real-world large-scale dataset collected across multiple cities in North
America.
To simulate the active learning setting, we follow standard practice in active learning research and treat the large labeled dataset as an unlabeled pool.
Active learning approaches select from this pool containing 100K scenes, with roughly 2 million labels.
For each scene, we have access to LiDAR sweeps recorded at 10Hz with a localized HD map given as input to the model.
For evaluation, we use the standard metrics of mAP@0.7 for detection and meanADE for prediction.
Implementation Details: For model implementation, we follow the exact details of the Gaussian mixture baseline from [12], an implementation of MTP [11] for the joint perception and prediction setting.
To support partial labeling, we ﬁnd it is necessary to use sum instead of mean to reduce losses in a batch, ensuring actors in less densely labeled scenes are not up-weighted relative to those in more densely labeled scenes.
All models are trained for 50 epochs, using budgeted training [40] for the learning rate schedule.
We use H = W = 20 to discretize the entire scene intorectangular regions for scoring and selection.
Finally, for sparsity regularization, we set M = 5 .
4.1 P&P from Partial Supervision
To test the effects of partial labeling, we randomly select labels at varying levels of ground-truth density per scene.
To ensure a fair comparison, the total labeling budget is ﬁxed across densities.
Speciﬁcally, let r be the labeling density.
When r = 1 , we recover the traditional fully labeled setting.
When r = 12 , we ﬁrst sample scenes randomly and then sample regions from half of each scene, providing labels only for selected regions.
Notice that since we ﬁx the labeling budget, selecting at lower densities r , will result in more scenes.
Figure 2 shows the effect of partial labeling on P&P performance.
Across all dataset sizes, we observe improved performance with lower density datasets.
Beneﬁts appear to saturate at r = 14 , as the sparser labeling density r = 16 does not provide further improvements.
Performance gains can be explained by the fact that more sparsely labeled datasets naturally include supervision from more scenes, improving the model’s ability to generalize.
Our results suggest adopting sparse labeling to optimize P&P performance under a ﬁxed labeling budget rather than label scenes entirely.
However, we note that in practice there is a tradeoff between labeling at lower densities and model training times, as datasets with less dense supervision must be trained for more iterations.
We explore this further in our sparsity regularization experiments.
Selected regions tend to have moving vehicles performing interesting actions (e.g. U-Turns)
4.2 Fine-Grained Active Selection for Prediction
Experimental Setup:
In this experiment, we study active learning approaches for improved prediction.
For each method, we sample an initial labeled set X (0) L of 40K vehicles from X U .
To ensure that ﬁnal datasets contain scenes with similar density of supervision, the initial data for ﬁne-grained methods are partially labeled scenes, whereas coarse-grained approaches sample full scenes.
For fair comparison, we ﬁx the labeling budget at each active learning iteration.
Speciﬁcally, for each of the N = 5 active learning iterations, the learner is given a budget of B = 20 K vehicles.
After each iteration, the model is re-trained on its current set of labels and evaluated on a seperate held-out test set.
The test-set is held constant across all approaches and contains traditional fully labeled scenes.
Additional Baselines: We compare ﬁne-grained active selection to full scenes selected randomly ( Random Scenes ), partially labeled scenes at density r = 14 selected randomly ( Random Regions ), full scenes selected by prediction entropy ( Coarse-Grained ), and two additional active learning baselines adapted to the P&P setting.
The ﬁrst is a recent uncertainty-based approach which learns to predict the loss of unlabeled examples, which we refer to as LearnLoss [1].
The second is the common diversity-based approach of Core-Set selection [21].
See the supplementary for details.
Prediction Performance: Results are shown in Figure 2.
All active selection techniques offer signiﬁcant improvements over random selection.
Interestingly, despite large differences in the selection criteria (e.g., uncertainty-based vs. diversity-based), scene-based approaches achieve similar performance, indicating that gains may be saturated due to the inﬂexibility of selecting entire scenes.
Surprisingly, simply labeling random regions appears to perform better than or similar to many of the coarse-grained active learning approaches.
Finally, ﬁne-grained selection offers the best performance.
While the improvements may appear relatively small, we ﬁnd consistent results across random seeds due to the large dataset sizes.
Additionally, aggregate prediction metrics are averaged over more than 1M actors in the test-set and may hide large differences between the speciﬁc behaviors of the prediction models, calling for more detailed analysis.
Performance By High Level Action: In Table 1, we break down the prediction performance by action: driving straight, turning left, turning right, and stationary.
We notice that differences between selection algorithms become more apparent across actions associated with more difﬁcult predictions (i.e., all non-stationary actions).
These results are explained by the fact that ﬁne-grained entropy selects more unpredictable moving actors compared to other selection approaches.
Planning Performance:
Following [12], we evaluate downstream performance on motion planning, computing the collision rate, L2 error, lateral acceleration, jerk, and progress of a planner [41] given the predictions from each trained model.
The results in Table 1 show that ﬁne-grained selection leads to a signiﬁcant reduction in collisions and also outperforms all baselines across the remaining planning metrics, except progress, for which the differences are not signiﬁcant (an obser-
[Figure]
S c e n e s w / L a b e l
Unconstrained
[Figure]
vation consistent with [12]).
The results demonstrate that the challenging training examples selected by active selection leads to improved planning, which is ultimately most important for self-driving.
Qualitative Examples and Selection Statistics: Examples of regions selected by ﬁne-grained selection, seen in Figure 3, tend to include vehicle labels with moving actors, actors at intersections, or actors performing odd maneuvers (e.g., U-Turn in the top-right example).
Non-moving actors are rarely selected.
In the supplementary materials, we plot histograms of label statistics selected by each method after the ﬁnal iteration of active learning.
As expected, we notice that active-selection methods tend to sample more non-stationary vehicles and vehicles further from the SDV.
This effect is more apparent for ﬁne-grained selection methods due to the additional ﬂexibility provided by the partially labeled setup.
Please see the supplementary for more detailed analysis.
Sparsity Regularization Ablation: In Figure 4, we ablate the effect of sparsity regularization on performance and number of scenes selected.
As expected, at early iterations, we ﬁnd unconstrained selection outperforms the sparsity regularized approach, as the unconstrained approach has the freedom to select a larger set of scenes, each with less supervision.
However, at later iterations, we observe the unconstrained selection performance degrades.
This is likely caused by the imbalance between the amount of supervision available for each scene, which we found empirically can lead to degraded performance.
Beyond performance, there is an additional, perhaps more important, beneﬁt of sparsity regularization.
Since the active learner must select at least M actors per scene, the number of scenes in the dataset grows linearly with each iteration.
Alternatively, in the unconstrained approach, the dataset size explodes at early iterations until there is a label for every scene in X U .
4.3 Fine-Grained Active Selection for Perception
We additionally experiment with ﬁne-grained selection for improved perception.
We follow a similar experimental setup, replacing prediction entropy with detection entropy.
As all methods perform similarly on vehicle detection, we evaluate on the more challenging multi-class setting where cyclists and pedestrians must be detected.
Similar to the prediction setting, results in Figure 4 show ﬁne-grained selection is most effective.
Interestingly, coarse-grained selection is similar to random scenes, likely explained by an averaging effect from summing the entropies over the full scene.
Conclusion
We studied active learning techniques to intelligently select examples to label from unlabeled selfdriving data logs for perception and prediction models.
We found the traditional active learning setting ill-suited and introduced generalizations to account for variable labeling costs and enable ﬁne-grained selection through partially labeled scenes.
In our experiments, we found signiﬁcant improvements from partial labeling without any active selection, and further gains across perception, prediction and downstream planning by leveraging ﬁne-grained active selection.
Our results demonstrate that the dominant paradigm of labeling entire self-driving scenes may not be most efﬁcient under a ﬁxed budget and that ﬁne-grained selection is likely required for maximal efﬁciency.