Just Label What You Need: Fine-Grained Active Selection for P&P through Partially Labeled Scenes
Anonymous Author(s)
AfﬁliationAddress email
Abstract: Self-driving vehicles must perceive and predict the future positions ofnearby actors to avoid collisions and drive safely.
A deep learning module is often responsible for this task, requiring large-scale, high-quality training datasets.
Due to high labeling costs, active learning approaches are an appealing solutionto maximizing model performance for a given labeling budget.
However, despiteits appeal, there has been little scientiﬁc analysis of active learning approaches forthe perception and prediction (P&P) problem.
In this work, we study active learning techniques for P&P and ﬁnd that the traditional active learning formulation isill-suited.
We thus introduce generalizations that ensure that our approach is bothcost-aware and allows for ﬁne-grained selection of examples through partially la- beled scenes.
Extensive experiments on a real-world dataset suggest signiﬁcantimprovements across perception, prediction, and downstream planning tasks.
Introduction
For self-driving vehicles to safely plan a route, they must perceive nearby actors and predict theirfuture locations.
In a self-driving stack, a learned perception and prediction (P&P) model is responsible for this task, taking raw sensor data as input and producing object detections and futurepredictions.
These models require large-scale, high-quality training datasets due to the high dimensional sensor inputs and long tail of future outcomes.
While self-driving companies collect massiveamounts of data from real-world driving, annotating the data remains a major bottleneck.
Furthermore, some of the data may be less interesting for model training – e g., a prediction dataset withmany parked vehicles is less informative than one with highly interactive, moving actors.
Therefore,the choice of which examples to label is crucial to maximize performance for a given budget.
Given a particular model, we seek to determine the examples most likely to improve performancewhen labeled.
This problem is well-studied in the ﬁeld of active learning and recent work has shownimpressive performance gains over random selection in many tasks, including image classiﬁcation,semantic segmentation and 2D object detection [1, 2, 3].
Active learning presents a promisingframework to employ in real-world self-driving development, where models can continually improveas new batches of examples are selected iteratively for labeling (see Figure 1).
Despite the appeal of active learning, few approaches have been developed for self-driving.
Scientiﬁc analysis is limited to object detection [4, 5], with no approaches designed for P&P.
Whenapplying active selection to this task, we ﬁnd the traditional formulation to be ill-suited.
First, whileapproaches typically assume ﬁxed labeling costs per example, annotation costs can vary drasticallyas they depend on the number of actors present.
Furthermore, the spatial label structure can be exploited to support partial labeling, enabling ﬁne-grained active selection.
Speciﬁcally, with smallmodiﬁcations, P&P models can be trained from partial supervision.
This allows the active learner toselect speciﬁc actors in a scene without requiring the remaining actors to be selected, as they may beuninteresting for model improvement.
With these differences in mind, we introduce a ﬁne-grained,cost-aware selection along with speciﬁc scoring criteria for active selection in the P&P setting
[Figure]
Params: K , N
Inputs: X U , X (0) L , M (model)
Outputs: X ( N ) L , M 1: for i ∈ 1 . . .
N do 2:
S ( i ) ← score( M , X U \ X
( i − 1) L ) 3:
Q ( i ) ← select top k
( S ( i ) , K ) 4:
X ( i +1)
L ← X ( i ) L ∪ Q ( i ) 5:
M ← train ( M , X
( i +1) L ) 6: end for
We leverage a real-world, large-scale dataset to analyze the effects of partial labeling and activeselection for P&P.
First, we study models trained on partial supervision, without active selection,and observe improved performance from sparser labeling.
Next, we analyze the added beneﬁtsof ﬁne-grained active selection and observe signiﬁcant gains compared to traditional approaches.
Further analysis shows the gains are most signiﬁant on rare events, translating to improvements indownstream planning performance.
All together, our analysis suggests that the dominant paradigmof labeling entire self-driving scenes is not the most efﬁcient use of a ﬁxed labeling budget and thatmore ﬁne-grained active selection may be required to most effectively select examples for labeling.
Related Work
Perception and Prediction: While perception and prediction have traditionally been handled separately, [6, 7] introduce models to jointly perform both tasks, improving performance and efﬁciency.
Among the exciting progress made in both tasks over recent years, most relevant to our work areimprovements in prediction representations, allowing models to better characterize uncertainty.
Examples of representations include trajectories [7, 8], probabilistic occupancy maps [9, 10], Gaussianmixtures [11], implicit latent variable models [12], and auto-regressive models [13].
Active Learning: We focus on pool-based active learning, in which new training examples arequeried from a large, unlabeled pool [14].
One class of approaches seeks to characterize modeluncertainty, measured via model disagreement [15, 16], entropy [17, 18], a learned loss prediction[1], or a discriminator score [19], and select examples with high uncertainty for labeling.
Whileoften effective, uncertainty based approaches are prone to selecting a subset of similar exampleswhen computational constraints require large batches of examples to be selected before retraining.
This motivates diversity-based approaches [20, 21, 22, 23] which seek to ﬁnd a representative subsetof the unlabeled pool.
[3, 24, 25] introduce approaches which balance both uncertainty and diversityin the selection process.
Most related to our domain are [4, 5, 26, 27, 28] which study active learningapproaches for object detection.
While most approaches assume ﬁxed labeling costs per example,[29, 30] have explored explicitly modeling individual labeling costs as part of the selection process.
Leveraging partially labeled data for ﬁne-grained active selection has been explored in semanticsegmentation [31, 32] and more generally, in the context of structured prediction problems [33].
Dataset Selection: Many self-driving datasets select examples manually [34], randomly, or viahardcoded rules.
Argoverse describes rules-based criteria to mine interesting trajectories for prediction [35].
More recently, [36] proposed a set of complexity measures for dataset selection.
Finally,[37] proposed tagging attributes of self-driving scenes, enabling retrieval for dataset curation.
Active Learning for P&P
Given the high labeling costs of P&P datasets, budgets can be spent more efﬁciently by intelligentlyselecting examples for labeling.
Active learning offers a promising solution, selecting examples believed most likely to improve model performance.
In this section, we review traditional poolbased active learning, where examples are iteratively selected from an unlabeled pool to build ahigh-quality labeled dataset.
Then, we address shortcomings in the P&P setting by introducing anew paradigm which is both cost-aware and enables ﬁne-grained selection through partially labeledscenes, providing the ﬂexibility to ensure budgets are spent effectively.
Finally, we provide concreteselection criteria used within our framework to optimize the model’s performance.
3.1 Traditional Active Learning
Self-driving companies can collect large-scale unlabeled real-world data when operating their vehicles.
Our goal is to select the best subset to label to improve model performance.
We assume accessto a large, unlabeled pool of examples, X U , and an initial subset of labeled examples, X (0) L .
Eachexample x ∈ X U represents an input to our model f ( x ) and if selected, a labeling oracle returns theground truth supervision, y
= L ( x ) .
In the P&P setting, inputs x represent raw sensor observationsand HD Maps, and labels y represent actor bounding boxes at the current timestep and for the prediction horizon of T seconds.
In each active learning iteration, we select a subset from the remainingunlabeled examples, Q ( i ) ⊂
X U \ X ( i − 1) L , query the labeling oracle, and add the examples to ourlabeled set.
With each iteration, the model is retrained or ﬁne-tuned with the latest dataset.
Traditionally, the active learner will select a ﬁxed number of examples at each iteration, | Q ( i ) | = K .
This implicitly assumes that each example x ∈ X U can be labeled for the same cost, an assumptionclearly violated in the P&P setting, which we will relax in the next section.
While a variety ofapproaches have been studied for active selection, we focus on methods which produce a scalar scorefor each example, S ( x ) ∈ R .
Scores represent some notion of informativeness where highly scoredexamples are believed to be most likely to improve model performance.
For example, measuresof model uncertainty, such as entropy, are commonly used (see Section 3.3 for concrete scoringfunctions for P&P).
As different models may beneﬁt from different types of examples, most scoringapproaches depend on the model’s current state.
After scores have been computed for the remainingunlabeled examples, the top K examples can be selected for labeling.
This process repeats for Nactive learning iterations and is summarized in Algorithm 1.
3.2 Fine-Grained Cost-Aware Active Learning
In this section, we generalize two aspects critical to the P&P setting, allowing for variable labelingcosts and ﬁne-grained selection through partial supervision.
Cost-Aware Active Learning: As a self-driving vehicle operates, the surrounding environmentwill change, leading to scenes with different labeling costs.
Crowded scenes can contain hundredsof actors, which are each traditionally labeled with a precise bounding box.
Sparser scenes, on theother hand, can be labeled with little manual effort.
To account for these differences, we explicitlymodel the cost to label each example, C ( x ) .
At each iteration, rather than select a ﬁxed K examples,the learner is instead given a ﬁxed budget B , which cannot be exceeded, (cid:80) x ∈ Q ( i ) C
( x ) ≤ B .
This formulation is a generalization of the previous setting, which can be recovered by setting
C ( x ) = 1 for all examples and B = K .
In practice, labeling cost for P&P examples can beaccurately modeled as a linear function of the number of actors in the scene as most annotation timeis spent drawing detailed bounding boxes for each actor.
This new formulation requires modiﬁcations to our selection algorithm, since high scoring examples S ( x ) may also have high costs C ( x ) .
Therefore, rather than sorting by score, we can select examples with the highest value, V ( x ) =
S ( x ) C ( x ) .
Since the cost of labeling a region is unknown before labeling, we approximate it using the numberof detections after non-maximum suppression (NMS) [38].
After labeling, the true cost is known,and active selection can continue iteratively until the budget is reached.
Since examples in the P&Psetting represent large scenes, scores S ( x ) and costs C ( x ) can vary signiﬁcantly as scenes can havefew to many actors, each contributing to the total score and cost.
As a consequence, coarse-grainedscoring will be suboptimal as scenes may contain regions with high score and low cost (e.g., a single car performing a rare U-Turn) and other regions with low score and high cost (e.g., a parking lotﬁlled with many static vehicles).
This motivates the need for more ﬁne-grained scoring and selection.
Next, we describe modiﬁcations to support partially labeled scenes, which will enable ﬁne-grainedselection for better performance in the cost-aware active learning setting.
Partially Labeled Scenes: We generalize the labeling process to allow for partial labeling.
Alongwith the added ﬂexibility, this setting is also realistic in practice.
Even as entire scenes are labeled today, annotation platforms often decompose work into subtasks, which can be more easily distributedand validated across a labeling team.
As a simple extension, platforms could support querying labelsfor only particular regions.
To support partial labels, we redeﬁne an example x R as the scene augmented with a labeling region R , x R = ( x , R ) .
Given the set of labels for the entire scene y
=
L ( x )and a region R , each actor’s bounding box label y i will either be fully contained in R , completelyoutside of R , or partially inside of R .
For simplicity, we assume that if any part of the bounding boxy i of an actor is inside R then it will be provided as a label.
In practice, this translates to labelersannotating all actors, even those that are only partially visible in the labeling region.
More formally,the labeling oracle returns labels for an example x R , L ( x R ) =
{ y i :
y i ∈
R and y i
∈
L ( x ) } .
Training from Partial Supervision: We adapt training to support partial supervision by applyingthe loss only on the labeled region, R .
Importantly, we do not alter the network input x , since wedo not want to bias the network by changing input statistics.
Therefore, the forward pass remainsunchanged, ˆ y = f
( x ) .
To compute the loss, we only consider the labels that we have received in R ,
[Equation]
Here, (cid:96) P ( · , · ) represents traditional multi-task perception and prediction losses applied over the positive examples in R and (cid:96) B ( R ) represents a “background” loss which encourages the network notto output detections for negative regions in R .
For example, in our experiments, (cid:96) P includes a probabilistic prediction loss, a bounding box regression loss and cross-entropy on positive examples,whereas (cid:96) B ( R ) represents the hard negative mining loss, sampling only negative anchors from R .
Fine-Grained Selection:
Without restrictions on R , there are inﬁnite regions to consider for agiven scene.
Therefore, in order to efﬁciently score and select regions for labeling, we consider theset obtained by discretizing the entire scene into a rectangular grid.
Speciﬁcally, we divide eachexample x into HW non-overlapping regions, x R = ( x , R h,w ) .
By setting H =
W
= 1 , weobtain a single region for each scene and recover the original formulation.
As H and W increase,candidate regions become smaller, providing the learner more ﬁne-grained precision for selection.
Most steps of the selection process can remain unchanged.
Scoring functions now operate overexamples augmented with regions, S ( x R ) returning a score that only considers network predictionsin R .
Similarly, only the cost of labeling the queried region C ( x R ) is incurred when selecting x R .
As regions sizes shrink, we observe that the active learner is more likely to select a large number ofscenes, each labeled with very sparse supervision.
While this dataset would contain many interesting actors, we also ﬁnd this unconstrained selection results in signiﬁcantly longer training times asexamples are less densely labeled.
Additionally, we observe training instabilities due to the imbalances between the amount of supervision available for each example.
To alleviate these issues, weintroduce a sparsity regularizer, which requires that the active learner select a minimum number ofpositive examples M for any selected scene.
Therefore, letting P ( x R ) represent the number of positive examples in an example, our new formulation can be summarized by the following optimizationproblem solved by the learner at each iteration,
[Equation]
We solve this optimization greedily by ﬁrst selecting the highest scoring scene remaining, thenselecting the highest regions in the scene until at least M actors are labeled.
We continue selectingnew scenes until the budget is reached.
[Figure]
t e r s ) r
=r = 1/ r = 1/
[Figure]
r
[Figure] [Figure]
3.3 Selection Criteria
Our ﬁne-grained, cost-aware active learning formulation introduced above generally supports anyapproach which provides an informativeness score S ( x R ) per example.
In this work, we focuson uncertainty-based approaches, an extremely common active learning paradigm based on the assumption that training on uncertain examples are most likely to improve future performance.
As thisapproach depends on a model’s characterization of uncertainty, we ﬁrst describe our probabilistic
P&P model, followed by possible uncertainty-measures that can be used for scoring.
Model: Following [6], we jointly train a model for both perception and prediction from LiDARand HD map inputs.
A model which naturally characterizes uncertainty over predictions is desirable,as these uncertainty estimates provide a useful measure of informativeness for scoring.
Therefore,we leverage the output representation of [11], using a mixture of K Gaussians to represent the distribution of each actor’s future positions.
For simplicity, independence is assumed between timestepsof the prediction horizon.
Thus the likelihood of a particular actor trajectory, y i , can be written as,
[Equation]
where N is the pdf of a 2D multivariate Gaussian with parameters µ tk , Σ tk , and π k represent Gaussian mixture weights.
These parameters, for each detected actor, are predicted by a deep neuralnetwork trained with negative log likelihood (see Section 4 for more details).
With knowledge ofthe model, we now introduce measures of uncertainty to use as selection criteria.
Due to the multitask nature of the task, we present separate selection criteria for the detection and prediction task.
Inpractice, a mix of both can be used to ensure performance improves across both tasks.
Detection Entropy: We focus on characterizing the uncertainty over the model’s classiﬁcationpredictions for each anchor.
For classiﬁcation tasks, the uncertainty is typically estimated by calculating the entropy of the model’s predicted probabilities.
Given anchors a ∈ A with associatedprobabilities p a , the entropy of the predictions are given by,
[Equation]
To score regions, we assume independence between anchors and sum the entropies of anchors in R .
Prediction Entropy: Computing prediction entropy naturally depends on the model’s output representation.
Our model outputs a Gaussian mixture for each predicted actor.
Unfortunately, thereis no known closed form solution to computing this distribution’s entropy [39].
Therefore, we arerequired to estimate the entropy via approximations.
We explored various approximations, includinga sample-based monte-carlo estimate, but all performed similarly to or worse than an approximationvia the entropy of the discrete categorical distribution induced by the mixture weights π k ,
[Equation]
Intuitively, this approximation is well-suited to capture cases where the model is uncertain betweenmultiple possible modes, which is likely representative of the true entropy of distribution.
We use
[Table]
this approximation due to its simplicity and computational efﬁciency while providing similar performance.
Finally, we assume independence and sum the actor entropies within each region.
Experiments
In this section, we analyze the effects of partial labeling and ﬁne-grained active selection.
First,we explore partial labeling independent of active selection.
Next, we explore the improvementsprovided by active selection for prediction.
We ﬁnd that a simple prediction entropy combined withﬁne-grained active selection outperforms various traditional scene-based approaches.
More detailedanalysis shows that ﬁne-grained selection enables the learner to better oversample labels exhibitingcomplex driving behaviors, resulting in better performance on these challenging behaviors in thetest-set.
In practice, we are most interested in the effects of these improvements on the downstreammotion planning task, where we ﬁnd signiﬁcant improvements across most metrics.
Finally, weobserve similar improvements for perception when using detection entropy as the selection criterion.
Dataset: We leverage a real-world large-scale dataset collected across multiple cities in North
America.
To simulate the active learning setting, we follow standard practice in active learningresearch and treat the large labeled dataset as an unlabeled pool.
Active learning approaches selectfrom this pool containing 100K scenes, with roughly 2 million labels.
For each scene, we haveaccess to LiDAR sweeps recorded at 10Hz with a localized HD map given as input to the model.
For evaluation, we use the standard metrics of mAP@0.7 for detection and meanADE for prediction.
Implementation Details: For model implementation, we follow the exact details of the Gaussianmixture baseline from [12], an implementation of MTP [11] for the joint perception and predictionsetting.
To support partial labeling, we ﬁnd it is necessary to use sum instead of mean to reducelosses in a batch, ensuring actors in less densely labeled scenes are not up-weighted relative tothose in more densely labeled scenes.
All models are trained for 50 epochs, using budgeted training[40] for the learning rate schedule.
We use H = W = 20 to discretize the entire scene into 400rectangular regions for scoring and selection.
Finally, for sparsity regularization, we set M = 5 .
4.1 P&P from Partial Supervision
To test the effects of partial labeling, we randomly select labels at varying levels of ground-truthdensity per scene.
To ensure a fair comparison, the total labeling budget is ﬁxed across densities.
Speciﬁcally, let r be the labeling density.
When r = 1 , we recover the traditional fully labeledsetting.
When r = 12 , we ﬁrst sample scenes randomly and then sample regions from half of eachscene, providing labels only for selected regions.
Notice that since we ﬁx the labeling budget,selecting at lower densities r , will result in more scenes.
Figure 2 shows the effect of partial labelingon P&P performance.
Across all dataset sizes, we observe improved performance with lower densitydatasets.
Beneﬁts appear to saturate at r = 14 , as the sparser labeling density r = 16 does not providefurther improvements.
Performance gains can be explained by the fact that more sparsely labeleddatasets naturally include supervision from more scenes, improving the model’s ability to generalize.
Our results suggest adopting sparse labeling to optimize P&P performance under a ﬁxed labelingbudget rather than label scenes entirely.
However, we note that in practice there is a tradeoff betweenlabeling at lower densities and model training times, as datasets with less dense supervision must betrained for more iterations.
We explore this further in our sparsity regularization experiments.
Selected regions tend to have moving vehicles performing interesting actions (e
-Turns
4.2 Fine-Grained Active Selection for Prediction
Experimental Setup:
In this experiment, we study active learning approaches for improved prediction.
For each method, we sample an initial labeled set X (0) L of 40K vehicles from X U .
To ensurethat ﬁnal datasets contain scenes with similar density of supervision, the initial data for ﬁne-grainedmethods are partially labeled scenes, whereas coarse-grained approaches sample full scenes.
Forfair comparison, we ﬁx the labeling budget at each active learning iteration.
Speciﬁcally, for each ofthe N = 5 active learning iterations, the learner is given a budget of B = 20 K vehicles.
After eachiteration, the model is re-trained on its current set of labels and evaluated on a seperate held-out testset.
The test-set is held constant across all approaches and contains traditional fully labeled scenes.
Additional Baselines: We compare ﬁne-grained active selection to full scenes selected randomly( Random Scenes ), partially labeled scenes at density r = 14 selected randomly ( Random Regions ),full scenes selected by prediction entropy ( Coarse-Grained ), and two additional active learningbaselines adapted to the P&P setting.
The ﬁrst is a recent uncertainty-based approach which learnsto predict the loss of unlabeled examples, which we refer to as LearnLoss [1].
The second is thecommon diversity-based approach of Core-Set selection [21].
See the supplementary for details.
Prediction Performance: Results are shown in Figure 2.
All active selection techniques offersigniﬁcant improvements over random selection.
Interestingly, despite large differences in the selection criteria (e.g., uncertainty-based vs. diversity-based), scene-based approaches achieve similarperformance, indicating that gains may be saturated due to the inﬂexibility of selecting entire scenes.
Surprisingly, simply labeling random regions appears to perform better than or similar to many ofthe coarse-grained active learning approaches.
Finally, ﬁne-grained selection offers the best performance.
While the improvements may appear relatively small, we ﬁnd consistent results acrossrandom seeds due to the large dataset sizes.
Additionally, aggregate prediction metrics are averaged over more than 1M actors in the test-set and may hide large differences between the speciﬁcbehaviors of the prediction models, calling for more detailed analysis.
Performance By High Level Action: In Table 1, we break down the prediction performance byaction: driving straight, turning left, turning right, and stationary.
We notice that differences betweenselection algorithms become more apparent across actions associated with more difﬁcult predictions(i.e., all non-stationary actions).
These results are explained by the fact that ﬁne-grained entropyselects more unpredictable moving actors compared to other selection approaches.
Planning Performance:
Following [12], we evaluate downstream performance on motion planning, computing the collision rate, L2 error, lateral acceleration, jerk, and progress of a planner[41] given the predictions from each trained model.
The results in Table 1 show that ﬁne-grainedselection leads to a signiﬁcant reduction in collisions and also outperforms all baselines across theremaining planning metrics, except progress, for which the differences are not signiﬁcant (an obser-
[Figure]
S c e n e s w / L a b e l
Unconstrained
Sparisty
[Figure]
vation consistent with [12]).
The results demonstrate that the challenging training examples selectedby active selection leads to improved planning, which is ultimately most important for self-driving.
Qualitative Examples and Selection Statistics: Examples of regions selected by ﬁne-grainedselection, seen in Figure 3, tend to include vehicle labels with moving actors, actors at intersections,or actors performing odd maneuvers (e.g., U-Turn in the top-right example).
Non-moving actors arerarely selected.
In the supplementary materials, we plot histograms of label statistics selected byeach method after the ﬁnal iteration of active learning.
As expected, we notice that active-selectionmethods tend to sample more non-stationary vehicles and vehicles further from the SDV.
This effectis more apparent for ﬁne-grained selection methods due to the additional ﬂexibility provided by thepartially labeled setup.
Please see the supplementary for more detailed analysis.
Sparsity Regularization Ablation: In Figure 4, we ablate the effect of sparsity regularization onperformance and number of scenes selected.
As expected, at early iterations, we ﬁnd unconstrainedselection outperforms the sparsity regularized approach, as the unconstrained approach has the freedom to select a larger set of scenes, each with less supervision.
However, at later iterations, weobserve the unconstrained selection performance degrades.
This is likely caused by the imbalancebetween the amount of supervision available for each scene, which we found empirically can lead todegraded performance.
Beyond performance, there is an additional, perhaps more important, beneﬁtof sparsity regularization.
Since the active learner must select at least M actors per scene, the number of scenes in the dataset grows linearly with each iteration.
Alternatively, in the unconstrainedapproach, the dataset size explodes at early iterations until there is a label for every scene in X U .
4.3 Fine-Grained Active Selection for Perception
We additionally experiment with ﬁne-grained selection for improved perception.
We follow a similar experimental setup, replacing prediction entropy with detection entropy.
As all methods performsimilarly on vehicle detection, we evaluate on the more challenging multi-class setting where cyclists and pedestrians must be detected.
Similar to the prediction setting, results in Figure 4 showﬁne-grained selection is most effective.
Interestingly, coarse-grained selection is similar to randomscenes, likely explained by an averaging effect from summing the entropies over the full scene.
Conclusion
We studied active learning techniques to intelligently select examples to label from unlabeled selfdriving data logs for perception and prediction models.
We found the traditional active learningsetting ill-suited and introduced generalizations to account for variable labeling costs and enableﬁne-grained selection through partially labeled scenes.
In our experiments, we found signiﬁcant improvements from partial labeling without any active selection, and further gains across perception,prediction and downstream planning by leveraging ﬁne-grained active selection.
Our results demonstrate that the dominant paradigm of labeling entire self-driving scenes may not be most efﬁcientunder a ﬁxed budget and that ﬁne-grained selection is likely required for maximal efﬁciency.