R4D: U TILIZING R EFERENCE O BJECTS FOR L ONGR ANGE D ISTANCE E STIMATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Estimating the distance of objects is a safety-critical task for autonomous driving.
Focusing on the short-range objects, existing methods and datasets neglect the equally important long-range objects.
In this paper, we name this challenging but underexplored task as Long-Range Distance Estimation, and propose two datasets for this task.
We then propose R4D, the ﬁrst framework to accurately estimate the distance of long-range objects by using references with known distances in the scene.
Drawing inspiration from human perception, R4D builds a graph by connecting a target object to all references.
An edge in the graph encodes the relative distance information between a pair of target and reference objects.
An attention module is then used to weigh the importance of reference objects and combine them into one target object distance prediction.
Experiments on the two proposed datasets demonstrate the effectiveness and robustness of R4D by showing signiﬁcant improvements compared to existing baselines.
I NTRODUCTION
[Figure]
Estimating the distances of objects from the vehicle is crucial for several autonomous driving tasks, including lane changing, route planning, speed adjustment, collision avoidance, to name a few.
Although existing methods and datasets focus on short-range objects, knowing the distance of long-range objects — objects beyond a typical LiDAR range of ∼ 80 meters (as shown in Figure 1) — is necessary for freeway driving, heavy-duty truck driving, and wet road driving.
Based on the US Department of Transportation (Blanco & Hankey, 2005), on rural highways with a standard speed limit of 65 miles/h, it takes ∼ 145 meters for a passenger vehicle to come to a complete stop in an emergency, greatly exceeding the typical LiDAR sensing range.
The required stopping distance grows signiﬁcantly with a heavy load truck or in bad road conditions such as snow, ice, or rain.
For example, the stopping distance increases from 145 meters to 183 meters and 278 meters for trucking and wet road driving, respectively (Administration, 2016; Blanco & Hankey, 2005).
In addition, given that harsh and sudden breaking on freeways is unsafe, it remains critical to estimate the distance of objects beyond the minimum required stopping distance in order to provide enough time for a gradual slow-down or lane change.
Therefore, to allow sufﬁcient time for an appropriate reaction and to ensure safety, autonomous driving systems are required to estimate the distance to long-range objects.
We name this critical but underexplored task as Long-Range Distance Estimation .
Concretely, given the short-range LiDAR signal and the camera image, the distances of long-range objects (beyond
LiDAR range) are expected as output.
Following conventions, the distance is measured between the camera and the object center along the camera’s optical axis (G¨okc¸e et al., 2015; Haseeb et al., 2018; Zhu & Fang, 2019).
We discuss more about making design choices for our task in the related work.
For this new task, we present pseudo long-range KITTI dataset and anonymous internal long-range dataset .
Since KITTI (Geiger et al., 2013) do not provide ground-truth distance for
Ref i
Ref N
Target
Ref 1 bbox, d bbox, d bbox, d ...
Raw Inputs
Feature Extraction d target
Tar-Ref Tar-Ref
i
Tar-Ref N
Embeddings ...
AttentionModule bbox long-range objects, the pseudo long-range KITTI dataset is a derived dataset by removing LiDAR points beyond 40 meters and considering the objects beyond 40 meters as target.
More importantly, we build a large scale dataset with annotated real long-range objects (from 80 to 300 meters).
In summary, with different deﬁnition of “long-range” (either beyond 40 or 80 meters), both datasets include LiDAR points, camera images, and distance labels for long-range objects.
Neither LiDAR nor camera alone solves this long-range distance estimation problem to the desired accuracy.
Most existing LiDAR technologies do not meet long-range sensing requirements.
According to the Waymo and KITTI self-driving car datasets, the maximum LiDAR range is only aroundmeters, which falls short of the range required for the aforementioned scenarios.
Even though some advanced LiDAR systems claim to achieve a longer sensing range, e.g ., Waymo’s 5th generation
LiDAR system (Jeyachandran, 2020) and Velodyne Alpha Prime TM (VelodyneLidar, 2021) which reach up to 300 meters, LiDAR points are sparse at long-range and, thus, more likely to be occluded.
Therefore, LiDAR alone is not enough to cover all safety-critical self-driving car applications.
Cameras , on the other hand, sense objects at a longer range and capture rich semantic information such as object appearance, geometry, and contextual hints.
However, they do not, in and of themselves, provide depth information.
Classical geometry-based algorithms could estimate the distance of canonical objects ( e.g ., sedans, trucks) based on their pixel sizes in the camera image (Criminisi et al., 2000; Tuohy et al., 2010; G ¨ okc¸e et al., 2015; Haseeb et al., 2018; Qi et al., 2019).
However, these approaches yield inaccurate results on long-range objects due to errors in size estimation.
Appearance-based methods (Song et al., 2020; Zhu & Fang, 2019) give unsatisfactory results on long-range objects.
Such methods rely on a single appearance cue to estimate the distance, overlooking context or other relevant signals in the scene.
At long-range, objects are visually small thus produce less informative appearance features.
Although neither LiDAR nor camera alone can solve this problem, these two signals provide complementary cues for long-range distance estimation.
In this paper, we propose R4D, a method to utilize r eference objects with known and accurate distances for long-range distance estimation (R4D).
The main motivation behind our approach lies in theories of human perception from cognitive science: humans often estimate the distance of an object relative to other reference points or objects (Granrud et al., 1984).
Speciﬁcally, we train a model to localize a long-range object ( target ) given references with known distances.
We represent the target object and references as a graph .
As shown in Figure 2, we deﬁne objects as nodes, with edges connecting the target to the reference objects.
The reference information is propagated to the long-range target object by extracting target-reference (Tar-Ref) embeddings.
R4D then feeds all target-reference embeddings to an attention module which fuses information from different references, by weighing their relative importance and combining them into one distance prediction.
Experiments on the pseudo long-range KITTI and anonymous internal long-range datasets show that
R4D signiﬁcantly improves the distance estimation and achieves state-of-the-art long-range localization results.
For example, on the anonymous internal long-range dataset, with R4D, the distances of 8.9% more vehicles (from 53.4% to 62.3%) are predicted with a relative distance error below 10%.
By conducting experiments on images captured at different times of the day, e.g ., train on daytime and test on nighttime images, R4D also shows stronger robustness against domain changes.
To summarize, our contributions are three-fold.
(1) We propose a critical but underexplored task
Long-Range Distance Estimation.
(2) We present two datasets, pseudo long-range KITTI dataset and anonymous internal long-range dataset.
To facilitate future research, the datasets will be made publicly available.
(3) We develop R4D, the ﬁrst framework to accurately estimate the distance of long-range objects by using references with known distances.
R ELATED W ORK
In this section, we discuss the related tasks and the detailed design choices of our proposed task.
Monocular distance estimation.
Estimating the distance of objects from an RGB image ( i.e ., monocular distance estimation) is a popular computer vision topic.
More than a decade ago, researchers designed geometry-based algorithms to estimate object distances.
For example, Tuohy et al.
(2010) use inverted perspective mapping (IPM) to convert an image to the bird’s eye view for estimating distances.
Learning-based methods were later proposed (Qi et al., 2019; Song et al., 2020).
For example, SVR (G¨okc¸e et al., 2015) and DisNet (Haseeb et al., 2018) take the pixel height and width of an object as input and estimate object distance by using support vector regression (Drucker et al., 1997) and multi-layer perceptron (MLP).
However, such methods are subject to errors in object size estimation (Zhu & Fang, 2019), which can be even more pronounced for longrange objects.
Zhu & Fang (2019) developed an end-to-end distance estimator with a Convolutional
Neural Network (CNN) feature extractor.
Speciﬁcally, given an RGB image and object bounding boxes, the model extracts per-object embeddings with a CNN and ROI pooling operator (Girshick, 2015), and then predicts a distance for each object.
As previously mentioned, this method relies heavily on object appearance, which is a less informative cue for long-range objects.
Another related research topic is monocular per-pixel depth estimation.
Eigen et al.
; Garg et al.
; Godard et al.
(2019); Lee & Kim (2019)
; Liu et al.
Shu et al.
(2020) propose to predict dense depth maps given RGB images.
For example, Lee & Kim (2019) generate multiresolution relative depth maps to construct the ﬁnal depth map.
However, these methods can be resource-intensive and, therefore, difﬁcult to incorporate within a latency-sensitive system such as autonomous driving (Zhu & Fang, 2019).
In addition, it is non-trivial to translate a depth map into per-object distance estimates due to occlusions, looseness of bounding boxes, etc .
Long-range depth sensing.
In addition, other approaches such as stereo cameras and Radar have been used to sense objects at long-range.
Stereo camera systems simulate human binocular vision and use multiple cameras to create stereo-pairs and perceive depth (Khamis et al., 2018;
Poggi et al., 2019).
However, stereo vision systems have a few limitations: they are difﬁcult to calibrate, are sensitive to vibrations (Zhang et al., 2020), and require a special hardware setup.
In contrast, R4D does not need special hardware and works with existing sensors present on most autonomous vehicles, such as standalone camera and LiDAR.
Radar, on the other hand, uses radio waves to determine the range, angle, and velocity of objects.
However, due to its limited angular resolution (Scheiner et al., 2020), it is difﬁcult for Radar to accurately localize long-range objects.
Monocular 3D object detection.
Monocular 3D object detection aims to detect all objects in an image by predicting a 3D bounding box for each of them, e.g ., MonoPair (Chen et al., 2020).
Besides distance (from the target object to the autonomous vehicle), the monocular 3D object detector also predicts object size.
For short-range vehicles, both object distance and object size are critical for selfdriving tasks, such as route planning, collision avoidance, and lane changing.
However, for longrange vehicles, the distance is signiﬁcantly larger than object size ( e.g ., 300 meters vs. 2 meters), thus the object size is less critical compared to the distance under the real-world scenario.
Moreover, predicting the object size increases the latency and the complexity of the model.
Therefore, our task focuses on estimating the distance of long-range objects.
[Table]
R# i Ebd.
T-R#1 Ebd.
T-R# n Ebd.
[Figure]
CNN
[Figure]
Concat
Aux MLP Δd
Aux MLP Δd n
Aux MLP Δd i … …
d targe t
Attention & MLP
[Figure]
M ETHODOLOGY
[Figure]
Motivated by the observation that humans estimate the distance of an object relative to other references (Granrud et al., 1984), we propose R4D, a method utilizing Reference objects For long-range Distance estimation.
References can be any combination of objects or points with known and accurate distances to the autonomous vehicle, such as LiDAR detections, objects detected by other sensors, map features, etc .
We represent the target object and references as a graph shown in the “Raw Inputs” dashed box in Figure 2.
The target object and its references are nodes in this graph.
Between a pair of target and reference objects, an edge is built to encode their pairwise relationship.
The detailed architecture is illustrated in Figure 3.
To model the target-reference pairwise relationship, we propose to extract union embeddings and geo-distance embeddings which encode visual and geometry relationships, respectively (Section 3.1).
Then, inspired by the intuition that reference objects are not equally important, we introduce an attention module in Section 3.2 to selectively aggregate pairwise relationships.
Finally, R4D is trained with an auxiliary supervision: the relative distances between the target and its references, as explained in Section 3.3.
In our setup, and without loss of generality, we use a monocular camera as the main sensor for detecting long-range target objects, and adopt LiDAR detected short-range objects as references.
It is worthy noting that R4D is not speciﬁcally designed for LiDAR or monocular images and is readily extended to other sensors and references.
3. M ODELING P AIRWISE R ELATIONSHIPS
As previously mentioned, to better estimate the distance of a target object, we propose to model the pairwise relationships between it and other reference objects.
For every pair of reference-target objects, we extract per-object feature embeddings and pairwise embeddings, as shown in Figure 3.
In particular, pairwise embeddings include union embeddings and geo-distance embeddings which provide visual and geometry cues, respectively.
Union embeddings.
Visual cues for modeling pairwise relationships should be based on the target object, the reference object, as well as the scene between the two objects.
We therefore propose a simple approach to extract feature embeddings from the union bounding box which covers both objects.
Figure 3 shows an example of a union bounding box highlighted in green.
This is similar to
Yao et al.
(2018), where union bounding boxes were used to extract features for image captioning.
Geo-distance embeddings.
To provide geometry cues, we form an input using the following:
- 2D bounding box center coordinates of target and reference objects, and the relative position shift between them. - 2D bounding box size of target and reference objects, and the relative scale between them.
- The distance of the reference object as provided by a LiDAR 3D object detector.
We, then, feed this input into a multi-layer perceptron to generate the geo-distance embedding.
Given the target, reference, union, and geo-distance embeddings, we concatenate them to form a ﬁnal target-reference embedding which models the relationship between a pair of target and reference objects.
We analyze and discuss the importance of these embeddings in Section 5.3.
3. A TTENTION -B ASED I NFORMATION A GGREGATION
Pairs of target-reference embeddings should be combined to estimate the distance of a given target object.
A simple approach is to average them over all references of the same target object.
However, intuitively, and as supported by our experimental results, reference objects are not equally important.
For example, a car in the bottom left of the image is potentially less helpful when localizing a faraway car lying in the top right corner of the image.
[Figure]
N u m b e r o f
V e h
i c l es
[Figure] [Figure]
In order to guide our model to focus on the most important reference objects, we introduce an attention-based module.
As shown in Figure 4, we follow VectorNet (Gao et al., 2020) to construct embeddings with local and global information.
Speciﬁcally, we use MLP and average pooling to extract a global embedding (shown in yellow in Figure 4).
The global embedding is, then, concatenated with the original target-reference embeddings (blue in Figure 4).
Given these global-local embeddings, the attention module predicts and normalizes importance weights for each reference using a fully connected layer followed by a softmax layer.
It, ﬁnally, uses these weights to fuse the original target-reference embeddings, in a weighted average manner, resulting in one ﬁnal embedding.
3. R ELATIVE D ISTANCE S UPERVISION
Predicting the target distance from the target embedding is more straightforward than from other indirect cues such as reference embeddings.
To encourage the model to learn the pairwise relationships between target and reference objects, instead of shortcut cues (Geirhos et al., 2020; Li et al., 2021;
Wang et al., 2020), we provide additional supervision.
The design of this additional supervision is akin to the residual representation, which is widely used for computer vision to help the optimization procedure (Briggs et al., 2000;
He et al., 2016; Szeliski, 1990; 2006).
Speciﬁcally, during the training stage, we add a relative (or residual) distance head for each target-reference embedding.
The relative distance ∆ d between the target object and the reference object is given by ∆ d = d t − d r , where d t (resp., d r ) is the distance between the camera and the target (resp., reference) object.
D ATASETS AND E VALUATION M ETRICS
To evaluate on the new task, long-range distance estimation, we create pseudo long-range KITTI dataset and anonymous internal long-range dataset .
We then introduce the evaluation metrics.
4. P SEUDO L ONG -R ANGE KITTI D ATASET
The original KITTI dataset (Geiger et al., 2013) contains RGB images, LiDAR point clouds, as well as 2D and 3D bounding boxes and other per-pixel annotations.
With these rich annotations, benchmarks for self-driving tasks have been well developed based on KITTI dataset, including scene ﬂow estimation, depth estimation, detection, and segmentation.
Even though the KITTI dataset provides LiDAR point clouds, it does not provide distance annotations beyond LiDAR range.
We hereby present pseudo long-range KITTI dataset , which is derived from the original KITTI dataset.
By assuming LiDAR has an effective sensing range of 40 meters only, we remove LiDAR points beyond it.
The objects lying beyond this effective sensing range are considered as long-range target objects, while the distance of other objects are known and provided as inputs.
It is worth noting that there are no overlap between the aforementioned target objects and the objects with known distance.
Following the convention (Shi et al., 2019), the original training data is split into two subsets for training and validation, respectively.
All the images do not contain long-range objects are removed from the dataset.
Finally, the pseudo long-range KITTI dataset contains 2,181 images with 4,233 vehicles and 2,340 images with 4,033 vehicles in the training and validation set, respectively.
Although it is easily obtained, this derived dataset is relatively smallscale and more critically does not annotate real long-range objects.
[Table]
A NONYMOUS I NTERNAL L ONG - RANGE D ATASET
Considering the aforementioned two limitations of the pseudo long-range KITTI dataset
[Table]
we build a long-range dataset which includes vehicle distances up to 300 meters from the autonomous vehicle.
The labels are obtained as follows.
First, we create 3D boxes for objects in the LiDAR range.
Then, we use Radar derived signals for rough positioning and extend LiDAR boxes to long-range.
Due to the low resolution of Radar, labelers use smooth object trajectory constraints to make sure the boxes are consistent over time and space.
Finally, given the 3D boxes for long-range objects, we compute their distances to the autonomous vehicle.
In total, as shown in Figure 5, we obtain 49, training images with 187,938 long-range vehicles, and 3,578 validation images containing 10, long-range vehicles.
These images span different times of day, including dawn, day, dusk, and night time.
Considering its large scale and the real long-range annotations, we use this dataset by default.
4. E VALUATION M ETRICS
We follow the evaluation metrics used in dense depth and per-object distance estimation (Eigen et al., 2014; Liu et al., 2015; Garg et al., 2016; Zhu & Fang, 2019; Shu et al., 2020; Zhang et al., 2020), namely: 1) the percentage of objects with relative distance error below a certain threshold ( < 5% ↑ , < 10% ↑ , < 15% ↑ ), 2) the absolute relative difference (Abs Rel ↓ ), 3) the squared relative difference (Sq Rel ↓ ), 4) the root of the mean squared error (RMSE ↓ ), and 5) the root of the mean squared error computed from the logs of the predicted and ground-truth distances (RMSE log ↓ ).
Speciﬁcally, given ground-truth distances d ∗ and predicted distances d , we compute the metrics as follows:
[Equation]
[Equation]
Note that ↑ and ↓ indicate whether a larger or smaller value of the corresponding metric indicates a better performance.
We only evaluate performance on long-range objects.
E XPERIMENTS 5. I MPLEMENTATION D ETAILS
We put the detailed design choices in Section A, and here we only introduce a critical component, distance augmentation, during the training stage.
Distance augmentation.
Distance augmentation is designed for encouraging R4D to learn from pairwise relationships and prevent it from exclusively using a single cue (namely, the target embeddings).
We guide the model to focus more on the relative distance between the reference and
[Table] [Table]
target objects by emphasizing the correlation between the pairwise embeddings and the distance prediction.
Speciﬁcally, we keep the relative distance ﬁxed, perturb the reference distance, and expect the model to predict the target distance with the same perturbation.
For example, as shown in
Figure 6 (a), the relative distance between the target and reference object is 120 meters and given a reference distance of 80 meters, the model should predict a target distance of 200 meters
(=meters + 120 meters).
As shown in Figure 6 (b), when the provided reference distance is perturbed (by, say, 20 meters), the model should predict 220 meters (= 100 meters + 120 meters) as the target distance, in order to maintain the correct relative distance of 120 meters.
This helps the model not overﬁt to appearance cues provided by the target object and be more robust to slight changes in camera parameters or ﬁeld-of-view.
Similar augmentation techniques have been successfully used to discourage shortcut learning in other deep learning tasks (Li et al., 2021).
During training, the distance label perturbation is sampled from a Gaussian distribution X ∼ N ( µ, σ 2 ) with µ = 0 and σ = 200 (See Section B for details).
For the pseudo long-range KITTI dataset, we use σ = 50 .
5. L ONG
[Figure]
ANGE D ISTANCE E STIMATION
[Figure]
Tables 1 and 2 show the distance estimation results on the anonymous internal long-range and pseudo long-range KITTI datasets.
On both datasets, R4D signiﬁcantly outperforms prior monocular distance estimation solutions.
For example, compared to Zhu & Fang (2019),
R4D signiﬁcantly improves the performance on the anonymous internal long-range dataset by 7.3%, 8.9%, and 8.4% on the < 5% ↑ , < 10% ↑ , and < 15% ↑ metrics, respectively.
We observe similar trends on the pseudo long-range
KITTI dataset: these improve by 6.9%, 6.7%, and 3.7%, respectively.
The improvement is even larger when compared to geometry-based techniques such as SVR (G ¨ okc¸e et al., 2015) and DisNet (Haseeb et al., 2018).
These improvements illustrate the effectiveness of R4D.
To further illustrate that R4D effectively uses references in LiDAR range for long-range distance estimation, we design another simple baseline to use LiDAR as a reference.
We augment the input camera image (RGB) with a channel containing per-pixel projected LiDAR distance values (D) (shown in Figure 1).
Then, given an RGB+D image, and detected bounding boxes, the model predicts a distance value for each target object.
We modify the input of our baseline, Zhu & Fang (2019), to include the distance channel (D), in addition to the RGB image.
However, as the results shown in Tables 1 and 2 (row 4), this approach achieves similar performance as when the camera image (RGB) is the sole input.
Besides, in Figure 7, we show the distance estimation results of R4D as well as Zhu & Fang (2019) with RGB and RGB+D inputs, broken-down over different groundtruth distance ranges.
We note that the addition of the LiDAR distance channel only improves the performance of objects slightly beyond LiDAR range, but fails at extrapolating distance values for localizing extremely long-range objects.
In contrast, R4D offers consistent performance gains compared to Zhu & Fang (2019) across all distances.
This results demonstrates the importance of explicitly modeling object relationships (the strategy of R4D), compared to the simpler baseline which has access to the same LiDAR sensor data.
[Table]
Results are slightly different from those
Tables 1 and 3 because, remove images which do not contain any reference objects.
This is in order to report results on
R4D-Relative, which requires at least one reference object (
Section
3 for details)
5. A BLATION S TUDIES
Attention-based information aggregation.
As discussed in Section 3.2, given a ﬁxed target, R4D aggregates all target-reference embeddings using an attention-based module and predicts a weight for each reference.
As shown in Table 1, compared to treating all references equally and taking the average of all target-reference embeddings (R4D-NA), the attention module improves the performance across all metrics, e.g ., < 5% ↑ is improved by 2.6%.
In order to inspect the attention module output, we visualize the learnt reference weights.
We highlight the reference object with the largest weight in Figure 8.
We ﬁnd that results match our intuition.
For example, the most important references usually share the same lane and pose as the target.
In addition, the best reference is usually the one closest to the target object.
Efﬁcacy of feature embeddings.
In this section, we perform ablation experiments on the proposed feature embeddings (Section 3.1) to show their importance.
The ﬁrst two rows in Table 3 show that, when only using the target and reference embeddings, the performance is sub-optimal.
This is likely due to the fact that these embeddings fail to capture pairwise relationships and only encode individual appearance, pose, and geometry information.
Incorporating the geo-distance embeddings (row 4) allows the model to learn from the relative position and geometry between target and reference objects, resulting in the largest performance improvement.
Finally, adding the union embeddings (row 5), which encode relative appearance cues, further improves the performance ( e.g ., < 5% ↑ improves by 1.6%).
Note that the union embeddings are only helpful when used together with the geo-distance embeddings.
This is because the union bounding box may include multiple potential objects, and a precise position is critical to pinpoint the reference object location.
Beneﬁts of modeling pairwise relationships.
R4D’s main performance improvements stem from modeling pairwise relationships between target and reference objects.
In this section, we show how incorporating references helps.
We study the effect of removing the relative distance supervision ( i.e ., the “Aux MLP” head in Figure 3) or the absolute distance supervision.
When only the relative distance head is enabled, we use its output during inference and the target distance is computed by adding the relative distance to the known reference distance (we exclude images without references).
[Table]
As shown in Table
[Table]
randomly choosing one reference (row 3) and predicting the distance using relative distance achieves a < 10% ↑ of 58.8%, outperforming the baseline (Zhu & Fang, 2019) which directly predicts the absolute distance (row 1 with a < 10% ↑ of 55.6%).
Combining the absolute and relative heads during training (row 5) further improves the < 10% ↑ metric by 4.3% (achieving a < 10% ↑ metric of 59.9%).
Finally, this improvement is the largest when multiple references are aggregated using the attention module (row 8), resulting in a < 10% ↑ of 63.3%.
Distance augmentation.
As described in Section 5.1, we propose the distance augmentation strategy to encourage R4D to learn better pairwise relationships during training.
Table 4 shows the results from ablation experiments related to distance augmentation.
The distance augmentation is only critical for our ﬁnal model R4D, which (1) includes both absolute and relative distance prediction heads, and (2) considers multiple reference objects.
Without distance augmentation (row 7), the model does not learn pairwise relationships, and the performance of R4D’s absolute distance head drops to the baseline’s performance (row 1).
In fact, when at most one reference is used during training (Zhu & Fang (2019), R4D-Relative, and R4D-One-Ref), the proposed distance augmentation marginally affects the performance.
Our intuition is that the one- or zero-reference cases require less supervision and data diversity due to the simpler input and model structures.
5. R OBUSTNESS ON O UT -O F -D ISTRIBUTION D ATA
For safety-critical applications, such as autonomous driving, the ability to handle out-of-distribution (OOD) data is crucial (Vyas et al., 2018).
We expect R4D to be more robust on OOD data because it captures contextual information, in addition to appearance and geometry cues.
In contrast, baseline methods (Zhu & Fang, 2019) use a single prediction cue, thus rendering them more sensitive to appearance changes.
To validate the robustness on OOD data, we train our model solely on images captured during daytime, but evaluate on three different times of day, namely: daytime, dawn/dusk, and night-time.
Speciﬁcally, we split the anonymous internal long-range dataset into four disjoint sets: train-day (used for training), and val-day , val-dawn&dusk , val-night (used for validation).
The results of this experiment are shown in Table 5.
Compared to the baseline method (Zhu & Fang, 2019), R4D achieves a larger improvement when the domain gap between the training and validation sets is larger.
For example, when evaluating on daytime (small domain gap) and night-time (large domain gap), R4D achieves an Abs Rel ↓ metric improvement of 1.6% and 5.6%, respectively.
C ONCLUSION
In this paper, we present an underexplored task, long-range distance estimation, as well as two datasets for it.
We then introduced R4D, a novel approach to address this problem.
Our method can accurately localize long-range objects using references with known distances in the scene.
Experiments on two proposed datasets demonstrate that it signiﬁcantly outperforms prior solutions and achieves state of the art long-range distance estimation performance.
It should also be noted that
R4D is not speciﬁcally designed for LiDAR or monocular images and is readily extended to other sensors and references, such as features from high-deﬁnition maps.
It can also be used to incorporate temporal information by fusing past object detections as references, which is a direction we hope to explore to further improve the long-range distance estimation performance.
A I MPLEMENTATION D ETAILS
We use ResNet-50 (He et al., 2016) followed by an ROIAlign layer (He et al., 2017) to extract target, reference, and union embeddings.
We then apply a fully connected layer to each embedding to reduce the feature dimension to 1,024.
We use a three-layer perceptron to extract geo-distance embeddings.
Each layer contains a single fully connected layer followed by layer normalization (Ba et al., 2016) and ReLU (Nair & Hinton, 2010).
For the attention module, we use the two-layer subgraph propagation operation mentioned in VectorNet (Gao et al., 2020) as the Global-Local Embedding Fusion depicted in Figure 4 in the main submission, and replace the max pooling layer with average pooling.
The reference objects are detected using a multi-view fusion 3D object detector (Zhou et al., 2020) with 72.1 3D AP (L1).
When there is no reference object in an image (0.6% of the images), we disable the relative distance supervision.
We use smooth L1 (Huber, 1992) as the loss function (Zhu & Fang, 2019).
We train R4D on an 8-core TPU with a total batch size ofimages for 24 epochs.
The learning rate is initially set to 0.0005 and then decays by a factor ofat the 16th and 22nd epochs.
We warm up (Goyal et al., 2017)
the learning rate for 1,800 iterations.
We initialize the weights from a ResNet-50 model pretrained on ImageNet (He et al., 2016).
For the pseudo long-range KITTI dataset, we train over 150 epochs with an initial learning rate of 0.005 and cosine learning rate decay (Loshchilov & Hutter, 2017).
[Table] [Table]
B H YPER - PARAMETER S ENSITIVITY : D ISTANCE A UGMENTATION
In this section
[Table]
we show how the performance of R4D varies with the distance augmentation hyperparameter σ .
In our ﬁnal model, we perturbed the ground-truth distances of target and reference objects by a Gaussian random variable with distribution X ∼ N ( µ, σ 2 ) with µ = 0 and σ = 200 .
In Table
[Table]
we show the performance of our method for different values of σ ranging from 1 to 300.
We observe that the proposed distance augmentation consistently improves the performance of R4D for all values of σ .
The model reaches its optimal performance when σ is around 200.
C I NFERENCE L ATENCY AND N UMBER OF R EFERENCES
In the main draft, we set the maximum number of references to 50.
In this section, we show the performance and inference time of our method when we vary the maximum number of references.
We draw two conclusions from the results (in Table 7).
First, more references lead to better results.
For example, we observe the with a maximum of 2 reference objects, the < 5% ↑ metric improves by 2% (from 35.0% to 37.0%) compared to using a single reference object.
In addition, when the number of references is large (say, 5), the beneﬁt of using additional reference objects is marginal.
For example, we observe that the < 5% ↑ metric increases by only 0.3% (from 37.2% to 37.5%) when the maximum number of references is increased from 5 to 50.
Finally, and as expected, the latency increases with the number of reference objects.
However, this increase is not signiﬁcant when using a relatively small number of references.
For example, when we increase the maximum number of references from 1 to 5, the latency only increases by 0.6 ms, while the < 10% ↑ metric improves by 9.2%.
D P SEUDO L ONG -R ANGE NU S CENES D ATASET
We further provide another derived pseudo long-rang dataset, Pseudo Long-Range nuScenes
Dataset , and the experimental results on it.
Although we prove several derived pseudo long-range datasets, we still encourage readers to focus more on our anonymous internal long-range dataset.
[Table]
[Table]
et al., 2020), and RTM3D (Li et al., 2020).
The proposed anonymous internal long-range dataset is a large-scale dataset with annotation for real long-range objects, which aligns better with the real-world scenario.
To construct pseudo long-range nuScenes dataset, we ﬁrst export the original nuScenes dataset to
KITTI format by the nuScenes ofﬁcal code 1 .
Then we reuse the pipeline that builds the pseudo long-range KITTI dataset.
The ﬁnal pseudo long-range nuScenes dataset includes 18,926 training images with target 59,800 vehicles, and 4,017 validation images with 11,737 target vehicles.
The results are shown in Table 8.
R4D outperforms all baselines with a large margin.
For example, with R4D, Zhu & Fang (2019) (line 3) improves by 4.4% on the < 10% ↑ metric (from 66.7% to 71.1%).
By comparing line 3 with line 4, we observe the performance drop when appending the
LiDAR points as an additional input channel to Zhu & Fang (2019).
This observation also supports our conclusion in the main manuscript, the additional distance channel does not help to improve the distance estimation performance.
The observation here is even more obvious, because the original nuScenes dataset only provides relative low resolution (32 beams, while KITTI dataset is 64 beams)
LiDAR signals (Caesar et al., 2019).
E C OMPARE WITH M ONOCULAR 3D O BJECT D ETECTORS
In the related work section of the main manuscript, we have discussed the relationship between our proposed task, namely Long-Range Distance Estimation, and monocular 3D object detection.
Here, we conduct experiments to compare R4D to monocular 3D detection methods on pseudo long-range
KITTI dataset.
We associate the detected 3D bounding boxes with 2D ground-truth boxes using a bipartite matching algorithm.
The weights of the bipartite graph are the 2D IoUs of ground-truth and detected boxes.
In order to compare with our method, we calculate the distance prediction metrics on matched ground-truth and detected box pairs.
As shown in Table 9, R4D outperforms SOTA 3D object detectors on the distance prediction task.
For example, on the < 5% ↑ metric, R4D improves 3.6% (from 42.7% to 46.3%) compared with previous state-of-the-art monocular 3D object detector RMT3D (Li et al., 2020).
This improvement is more signiﬁcant than the improvement from SMOKE (Liu et al., 2020) to RTM3D (Li et al., 2020) (only improves 0.3% on the < 5% ↑ metric) due to R4D utilizing the short-range LiDAR signals.
It is also worth noting that R4D does not utilized the techniques proposed in other monocular 3D object detection papers such as multi-task training and strong architectures.
Therefore, R4D is complementary to 3D detection methods, and combining it with such methods will further improve the long-range distance estimation performance.
[Table]
F M ORE Q UALITATIVE E XAMPLES ON A NONYMOUS I NTERNAL L ONG - RANGE D ATASET
We also provide more qualitative examples of the output of the attention module in the supplemental material (please download the zip ﬁle).
All images are from our proposed anonymous internal longrange dataset.
Similar to Figure 8 in the main manuscript, the target objects are highlighted in yellow.
References with the largest attention weights are highlighted in red.
Other references are colored in blue.
As before, we observe that the most important references are often close to or in the same lane as the target, and share a similar pose.
G U NDERSTANDING D ISTANCE A UGMENTATION
In this section, we illustrate that the baseline model (Zhu & Fang, 2019) trained with or without distance augmentation should achieve the same objective.
Intuitively, since the distance augmentation we used has zero mean ( µ = 0 in the Gaussian distribution), the models predict unbiased distance because the model tries to minimize the loss for all examples.
Given the same input image, the N ground-truth labels (in N epochs) are different due to distance augmentation.
The model can only predict one value because the input image is ﬁxed and tend to minimize over all losses between the only one prediction and all N labels.
To achieve the minimal loss, the model predict the mean value over N labels (if using L2 loss function) or the median value over N labels (if using L1 loss function).
When N is large, the mean/median value is close to the original label.
To summarize, given the same input image, if simply perturbing the label using
Gaussian noise, the model will attempt to predict the mean/medium so that the loss is minimized.
Our observation from the ﬁrst two lines in Table 4 also support our intuition: the performance of the baseline models (Zhu & Fang, 2019) is similar no matter with or without distance augmentation.
H A BLATION S TUDIES ON O UT -O F -D ISTRIBUTION D ATA
In Section 5.4, we demonstrate the robustness of R4D on Out-Of-Distribution data.
In this section, we study the impact of each component of our proposed R4D on anonymous internal long-range.
The results are shown in Table 10.
The observation is similar to our observation on the standard validation set (Section 5.3): each component incurs marginal improvement on model robustness, and only when using all of these components together, a signiﬁcant improvement can be observed.
For example, before applying Geo-distance embedding, the improvement on Night set is only 0.7%
Abs Rel ↓ , while the ﬁnal R4D model improves 5.6% Abs Rel ↓ .
Another interesting observation is about distance augmentation: the model without distance augmentation does not perform well on the Day Set, but perform as well as the model with distance augmentation on the Night Set.
We hypothesis this is because the appearance information for night scene are not discriminative enough, so both models rely more on the geometry information.
We will further investigate this in the future.
I R OBUSTNESS A GAINST R EFERENCE B OXES P ERTURBATION
The goal of this section is to demonstrate how robust R4D is with respect to the noise of the input 2D bounding boxes and the object distance provided by the 3D detector.
We conduct ablation studies by perturbing the 2D reference boxes (with a maximum magnitude of 15% of the box size) or the distance of the reference objects (with a maximum magnitude of 15% of the original distance).
The results are shown in Table 11 and Table 12.
We found that even with signiﬁcant noise ( i.e ., 15%), the performance of our model still signiﬁcantly outperformed our baseline model (Zhu & Fang, 2019).
For example, in Table 12, even though with 15% noises, R4D is still 4.3% < 15% ↑ better than the baseline method (Zhu & Fang, 2019).
[Table]
the performance of
Although slightly observing drop after adding noise
R4D still better than the baseline method (Zhu & Fang, 2019)
[Table]
[Table]
on the short range objects, and then predict the distance for short-range objects as references.
J C OMPARE WITH B ASELINE WITH A P URE I MAGE - BASED S ETTING
To demonstrate the potential of the concept of using reference object for depth estimation, in this section, we conduct experiment in a purely image-based setting in which the depth for of the reference object is generated by a monocular depth estimation model.
Speciﬁcally, we use the monocular method proposed by (Zhu & Fang, 2019) to generate short range distance (with 8.14% absolute relative error for short-range objects).
As shown in Table 13, in a purely image based setting, our model signiﬁcantly outperforms the baseline models.
The results conﬁrms the effectiveness of our purposed reference-based method.