Vision-based Uneven BEV Representation Learning with Polar Rasterization and Surface Estimation
Abstract:
In this work, we propose PolarBEV for vision-based uneven BEV representation learning.
To adapt to the foreshortening effect of camera imaging, we rasterize the BEV space both angularly and radially, and introduce polar embedding decomposition to model the associations among polar grids.
Polar grids are rearranged to an array-like regular representation for efﬁcient processing.
To determine the 2D-to-3D correspondence, we iteratively update the BEV surface based on a hypothetical plane, and adopt height-based feature transformation.
PolarBEV keeps real-time inference speed on a single 2080Ti GPU, and outperforms other methods for both BEV semantic segmentation and BEV instance segmentation.
Thorough ablations are presented to validate the design.
The code has been released for facilitating further research at https: //github.com/ SuperZ-Liu/PolarBEV .
Introduction
Bird’s Eye View (BEV) representation [1, 2, 3, 4, 5, 6, 7] is of great practical value for environmental perception in autonomous driving.
Especially for vision-based system, BEV implicitly and elegantly aggregates multi-view information into a uniﬁed representation, avoiding time-consuming post processing for multi-view fusion.
This work proposes PolarBEV for vision-based uneven BEV representation learning.
We rasterize the BEV space both angularly and radially, making BEV grids densely distributed near the egovehicle and sparsely distributed far from the ego-vehicle, i e. , distance-dependent uneven grid distribution.
Recent works [2, 8, 9] rasterize the BEV space along the cartesian axes and get evenly distributed rectangular grids.
Such rectangular BEV representation is straightforward, but polar
BEV representation makes more sense.
First, for a self-driving car, the concerned perception region is centered at the ego vehicle.
Surrounding perception results are more important than distant ones for avoiding trafﬁc accidents.
Thus, higher resolution in surrounding areas is expected.
Second, for even BEV representation, long-range BEV space ( e.g. , 100 m × 100 m ) requires a large number of BEV grids and high computational budget.
Polar rasterization enables long-tailed uneven grid distribution, which can be ﬂexibly adjusted to cover large BEV space with limited computation cost.
We assign angle-speciﬁc and radius-speciﬁc embeddings to each polar grid according to its 3D position.
Because of the foreshortening effects of camera imaging, object’s scale in image varies a lot when the distance to camera changes.
Polar grids at the same distance correspond to the same scale.
Grids at the same angle correspond to the same camera view.
With angle-speciﬁc and radiusspeciﬁc embeddings, we model the associations among grids to enhance the BEV representation.
We propose iterative surface estimation for effective and efﬁcient BEV representation learning.
Previous methods [2, 9] usually predict pixel-wise depth distribution and broadcast pixel features to BEV space.
Differently, we ﬁrst set a hypothetical BEV surface and iteratively update the height of each polar grid to adjust the 2D-to-3D correspondence between image pixels and BEV grids.
Height is much easier to be estimated than depth.
The iterative reﬁnement process leads to more precise 2D-to-3D feature transformation and better BEV representation.
PolarBEV achieves real-time inference speed ( 25 FPS on a 2080Ti GPU), and signiﬁcantly outperforms counterparts for both BEV semantic segmentation and BEV instance segmentation.
Given the high efﬁciency and strong performance, PolarBEV can be integrated into autopilot system for online environmental perception.
Related Work
Recently, many large multi-sensor datasets [10, 11, 12] made it possible to directly supervise models by projecting 3D annotations onto the ground plane to generate BEV labels.
The key to the problem is how to model the transformation from image view to Bird’s Eye View, which is inherently an ill-posed problem.
A straightforward method is to assume the world is ﬂat and transform the image to BEV map through Inverse Perspective Mapping (IPM) [1, 13, 14].
Though this approach works in some cases, it often introduces artifacts to objects that lie above the ground plane.
In order to achieve better results, other methods [15, 16, 17] explicitly estimate depth to lift objects into BEV.
And OFT [18] maps image-based features into an orthographic 3D space with the aid of camera parameters.
A potential performance bottleneck of this method is that the contribution of each pixel feature is independent of objects depth at that pixel.
Instead of copying each pixel feature along camera ray, Lift-Splat [9] learns a depth distribution for each pixel.
Recently, FIERY [2] has extended Lift-Splat [9] further to use multi-timestamp observations for motion forecasting.
Different from these methods, we adopt BEV surface estimation instead of depth distribution to determine the correspondence between image and BEV.
Another technical route directly predicts BEV outputs from input images.
CVT [8] encodes the camera parameters into positional embeddings to model the geometric structure of the scene implicitly.
Different from CVT [8] using global attention to update each query, GKT [7] leverages the geometric priors to guide the transformer to focus on discriminative regions.
VED [6] predicts a semantic occupancy grid directly from the front-view image with a variational encoder-decoder network.
VPN [19] proposes a fully-connected view relation module to predict the semantic BEV map from multiple views.
PON [20] further advances fully-connected layer for each column to translate features from image space to BEV space.
Instead of using fully connected layers, TIM [21] models the relation of image columns and BEV polar rays with cross-attention.
Based on TIM [21], the work [22] further employs a graph network to spatially reason about an object within the context of other objects.
BEVFormer [23] predeﬁnes a set of uniformly distributed height anchors and projects these anchor points to image to get features.
In 3D domain, [24, 25] divide the 3D space into polar grids for point cloud segmentation, in order to adapt to the long-tailed distribution of LiDAR points. [26] leverages the radial symmetry to normalize point cloud along the radial direction.
[27] introduces polar parametrization for 3D detection to establish explicit associations between image patterns and prediction targets.
Differently, considering the foreshortening effects of camera imaging and the characteristics of BEV perception, we adopt polar rasterization for vision-based BEV representation learning.
Method 3.1 Overview
The framework of PolarBEV is presented in Fig.
Taking multi-view images as input, we ﬁrst extract image features with shared CNN backbone.
We rasterize the BEV space along the polar coordinates and rearrange the polar grids to array-like regular representation.
Then, we iteratively update the BEV surface with grid-wise height estimation, and transform 2D features to BEV features based on the estimated height and camera’s calibrated parameters.
BEV surface updating and feature transformation are repeated in a cascade manner for several times.
And various heads follow the ﬁnal
BEV representation to perform BEV perception.
Detailed designs are presented below.
[Figure]
Circular Padding
Rearrange
[Figure]
Hypothetical BEV Surface
Updated BEV Surface 𝐷 𝑎𝑛𝑔 𝐷 𝑟𝑎𝑑 𝑅 𝑚𝑎𝑥
[Figure]
Array-like Polar Representation Height
[Figure]
Height
[Figure]
Projection
Grid
[Figure]
3.2 Polar BEV Rasterization and Rearrangement
The concerned BEV space is centered at the ego-vehicle with the radius R max and 360 ◦ FoV (ﬁeld of view) coverage.
As shown in Fig.
1, we rasterize the BEV space both angularly and radially.
Radially, we evenly divide [0 , R max ] into D rad segments ( R max and D rad respectively denote radius maximum and radial resolution).
Angularly, we evenly divide the 360 ◦ into D ang segments ( D ang denotes the angular resolution).
After rasterization, for efﬁciently processing the polar representation, we rearrange polar grids along the angular and radial dimensions and get array-like regular representation with shape D rad × D ang .
The rearranged representation is hardware-friendly but can not be processed by conventional convolution operation.
As shown in Fig. 1, in the angular dimension, − π and π correspond to the the same angle but are separated in the array-like regular representation.
Alternatively, we adopt ring convolution [25] with circular padding to process the rearranged representation.
Speciﬁcally, we ﬁrst circularly pad in the angular dimension, and then adopt conventional convolution with 0 padding in the radial dimension.
The ring convolution is followed by batch normalization and ReLU activation layers.
3.3 Polar Embedding Decomposition
For each polar grid with radial distance r and angle θ , we predeﬁne a learnable query embedding q and decompose q into two components, i.e. , radius-speciﬁc query q ( r ) rad and angle-speciﬁc query q ( θ ) ang , which is formulated as,
[Equation]
For the polar representation, polar grids at the same distance correspond to the same scale, and grids at the same angle correspond to the same camera view.
With angle-speciﬁc and radius-speciﬁc embeddings, we model the associations among grids to enhance the BEV representation.
Ablation experiments are presented in Sec.
4.3 to validate the effectiveness of embedding decomposition.
3.4 Iterative Surface Estimation and 2D-to-3D Feature Transformation
We perform 2D-to-3D feature transformation based on a height-based projection mechanism.
To make sure the correspondence between image and BEV, we ﬁrst set a hypothetical BEV surface with height h hypo and then iteratively update the height of each grid i based on its query embedding q i .
This can be formulated as follows:
[Equation]
where Θ is a MLP layer, t is the iteration index and h
i is the hypothetical height h hypo .
Then we normalize h i (superscript t is omitted for clarity) to the range of [0 , 1] with sigmoid function σ and further scale up the value to the range of [ Z inf , Z sup ] , i.e. ,
[Equation]
where Z sup and Z inf are the predeﬁned upper and lower bounds of height respectively.
Each grid i corresponds to a polar coordinate p i =
( r i , θ i ) (the position of the grid’s center point).
We ﬁrst transform each polar coordinate p i to cartesian coordinate ( x i , y i ) in the following manner:
[Equation]
Then we concatenate the cartesian coordinate ( x i , y i ) with its corresponding height
z i to construct the homogeneous coordinate w i , i.e. ,
[Equation]
[Equation]
where ⊕ denotes the concatenation operation.
Then we project w i to the image plane with camera’s intrinsics I ∈ R K × 3 × 3 and extrinsics E ∈ R K × [Equation]
[Equation]
where u i,n is the projected image point on the n th view of BEV grid i , and K represents the total image views.
Finally, we transform features from the image view to the Bird’s Eye View with the projected coordinates.
This can be formulated as:
[Equation]
where F i,n is the sampled image feature corresponding to the projected image point u i,n , and M n is a binary mask for masking out projected points which exceed the image boundary.
After that, f i is sent into a MLP layer and its output is added to the previous query for query update.
The surface estimation and 2D-to-3D feature transformation are conducted for several times and the polar representation is iteratively updated.
3.5 Segmentation Head
We follow the head design of FIERY [2].
We adopt a small encoder-decoder network to further reﬁne BEV features.
Three branches follow the network, respectively for predicting segmentation score, offset and centerness.
The loss settings are also the same with FIERY [2].
Polar predictions are mapped into rectangular predictions for loss calculation based on the rectangular ground truths.
Experiments 4.1 Experimental Settings
Datasets NuScenes [10] dataset is a large-scale autonomous driving dataset which was collected over a variety of weathers and trafﬁc conditions.
This dataset contains 1000 scenes and each scene lasts 20 seconds.
The captured RGB images are from six cameras covering a full of 360 ◦ around the ego-vehicle.
Each camera has calibrated intrinsics and extrinsics at every timestamp.
Following common practice [2, 8], we project the 3D box annotations of vehicles onto the ground plane to get ground truth labels.
Architecture We adopt the pre-trained EfﬁcientNet-B4 [28] as backbone to extract multi-view image features.
Features of stride 8 are taken as the input for feature transformation.
The initial polar BEV feature is a combination of angular queries and radial queries with the shape [-of
-]of 400 × 100 ×for representing a certain BEV area (setting 1 or setting 2) centered at the ego-vehicle.
We set the radius maximum R max as the distance from the farthest point of BEV area to the center of egovehicle.
The small encoder-decoder network contains ﬁrst four layers of ResNet-18 [29], and all the convolution layers with kernel size greater than 1 are changed into ring convolutions to adapt to the polar representation.
[Table]
[Table]
Training All our networks are implemented with Pytorch [30] and Pytorch Lightning.
The input images are ﬁrst resized and randomly cropped into a special resolution before being fed into the networks.
The AdamW [31] optimizer is used for training the networks, with weight decay of 1 e − 7 .
We apply the one-cycle learning rate scheduler [32] to adjust the learning rate.
All networks are trained on 8 GPUs with a batch size of 16 .
Evaluation Two settings are used for evaluating vehicle segmentation map.
Setting 1 [20] perceives a 100 m × 50 m area around the ego-vehicle with 0 .
25 m resolution, while the perceptual range of setting 2 [9] is 100 m × 100 m with 0 .
5 m resolution.
These two settings serve as the main comparisons to prior works.
We use setting 2 for all the ablation studies.
The Intersection-OverUnion (IoU) metric is used for evaluating vehicle BEV segmentation performance for both settings.
We also report Panoptic Quality (PQ) metric for evaluating vehicle BEV instance segmentation performance, following previous work [2].
Additionally, we report the inference speeds measured on a single 2080Ti GPU.
The number of iterations and input resolution for PolarBEV are set to 3 and 224 × 480 respectively for most experiments unless speciﬁed otherwise.
Results of ablation studies are produced with masking invisible vehicles.
4.2 Performance Comparison
To validate the effectiveness and efﬁciency of the proposed method, we compare PolarBEV with competitive approaches on both BEV semantic segmentation and BEV instance segmentation.
All the methods involved in comparison only use single frames as input without temporal information.
BEV Semantic Segmentation For fair comparison, in Tab. 1 and Tab. 2, we respectively compare results without and with masking invisible vehicles.
Tab. 1 compares results without masking invisible vehicles.
Our model with 2 iterations outperforms counterparts by a signiﬁcant margin in both settings, especially for 448 × 960 input resolution.
Even compared with the best prior method FIERY Static [2], our method ( 224 × 480 input) is still 3 .
8 points higher in setting 1 and performs 3 × faster inference with the same parameters.
When the input resolution is increased to 448 × 960 , our model
[Table] [Table] [Table]
achieves further 7 .
7 points higher in setting 1 but still runs in a faster speed than FIERY Static.
In Tab. 2, we compare PolarBEV with other methods with masking invisible vehicles.
Because there is no ofﬁcial results of FIERY Static [2] in this setting, we reproduce it with the ofﬁcial code.
As this table shows, our model ( 224 × 480 input) with 2 iterations also achieves the best results in both settings and achieves real-time performance.
BEV Instance Segmentation We also provide BEV instance segmentation comparison with masking invisible vehicles in Tab. 3.
The results of FIERY Static [2] are reproduced with the ofﬁcial code.
As shown in this table, our model with 2 iterations is more than two points higher than FIERY Static in both settings in terms of PQ.
We also report Recognition Quality (RQ) and Segmentation Quality (SQ) in this table.
As we can see from this table, the revenue mainly comes from RQ, meaning our model detects instances more accurately.
4.3 Ablation Study
Polar vs. Rectangular Grid Distribution In Tab. 4, we present ablation experiments about polar and rectangular grid distribution.
For fair comparison, both results have the same number of BEV grids, 200 × 200 for rectangular grid distribution and 400 × 100 for polar grid distribution.
We keep the network structure the same.
As shown in Tab. 4, polar grid distribution achieves an improvement of 0 .
53 IoU and 0 .
33 PQ with no degradation in inference speed.
Height-based vs. Depth-based Feature Transformation In Tab. 5, we provide ablation experiments about feature transformation.
FIERY Static [2] adopts depth-based transformation, thus we directly take it as baseline for comparison.
To get the height-based results, based on FIERY Static, we change the feature transformation manner to the one introduced in Sec 3.4.
Tab.
5 shows that height-based results surpass depth-based results a lot in both IoU and PQ with nearly 3 × inference speed, validating the advantage of height-based transformation.
The advantage mainly comes from the following points.
The numerical range of depth is [0 , + ∞ ) for each pixel, which is hard for the network to estimate a reasonable value in such a huge space. While Ring Convolution [Table]
[Table] [Table] the numerical range of height for BEV is quite small.
Height-based transformation corresponds to less prediction error.
As for the speed, depth-based methods (such as FIERY Static) broadcast pixel features to BEV space along the depth dimension and sum all the 3D features along the vertical dimension.
These operations are time-consuming.
While height-based transformation is lightweight, resulting in less latency.
Ring Convolution and Polar
Embedding Decomposition In Tab. 6, we ablate on the ring convolution and the proposed polar embedding decomposition.
The baseline result achieves 40 .
48% IoU and 34 .
39% PQ.
After adopting ring convolution, we observe an improvement of 0 .
33 in IoU and 1 . 03 in PQ respectively.
In order to model the associations among polar grids, we decompose grid embeddings q into angle-speciﬁc ones q ang and radius-speciﬁc ones q rad .
As the last row shows, embedding decomposition further brings an improvement of 0 .
55 IoU and 0 .
36 PQ.
Polar BEV Resolution To verify how the resolution of polar BEV representation affects model performance
, we performe ablation experiments on the angular resolution and the radial resolution respectively.
As shown in the upper of Tab. 7, the model performance becomes better with the increase of angular resolution and reaches saturation at 400 .
The same rule can be found in radial resolution, as the radial resolution increases, the results become better.
Although increasing the angular resolution or the radial resolution brings beneﬁts and barely increases model parameters, the memory occupation is proportional to the angular or radial resolution.
Considering both the performance and memory occupation, we choose the resolution of 400 × 100 as the default setting.
The Number of Iteration We also conduct ablation study about the number of BEV surface estimation iterations in Tab. 8.
As this table shows, our model with only 1 iteration gets 40 .
73% IoU and 34 .
59% PQ, which has already exceeded all previous methods by a signiﬁcant margin but with a real-time inference speed.
When the number of iteration is 2 , the method achieves higher performance in both IoU and PQ metrics and still keeps real-time speed.
After adding the number of iteration to 3 or 6 , the performance seems to reach saturation, while the FPS degrades a lot when the number of iterations is 6 .
4.4 Qualitative Results
Fig.
2 shows several qualitative results on various scenes.
We give the six camera views and the predicted segmentation results and instance segmentation results along with the ground truth in each row.
As can be seen from this ﬁgure, PolarBEV can accurately segment vehicles in a variety of complex scenes.
4.5 Limitations
We present some failure examples in Fig.
As shown in the upper of this ﬁgure, it is hard for
PolarBEV to predict the exact location when vehicle is far away from the ego-vehicle.
As the bottom of this ﬁgure shows, PolarBEV fails when the height of vehicle is hard to estimate because of blocking.
In order to achieve better results in the future, we may consider using temporal information to calibrate the failures.
Conclusion
We propose PolarBEV for vision-based uneven BEV representation learning in this work.
PolarBEV rasterizes the BEV space both angularly and radially, making a distance-dependent uneven grid distribution.
To model the associations among grids, we assign angle-speciﬁc and radius-speciﬁc embeddings to each polar grid.
Different from previous depth based methods, we ﬁrst set a hypothetical BEV surface and then iteratively update the height of each polar grid to adjust the 2D-to-3D correspondence between image pixels and BEV grids.
Extensive experiments show PolarBEV is an alternative way for better and faster segmentation. Acknowledgments
This work was supported by the National Natural Science Foundation of China under Grant no.