Terrain Traversability Prediction for Off-Road Autonomous Driving
Anonymous Author(s)
AfﬁliationAddress
Abstract:
In this paper, we focus on the problem of estimating the traversability of terrain for autonomous off-road navigation.
To produce dense and accurate local traversability maps, a robot must reason about geometric and semanticproperties of the environment.
To achieve this goal, we develop a novel Bird’s
Eye View Network (BEVNet), a deep neural network that directly predicts densetraversability maps from sparse LiDAR inputs.
BEVNet processes both geometricand semantic information in a temporally consistent fashion.
More importantly, ituses learned prior and history to predict traversability in unseen space and into thefuture, allowing a robot to better appraise its situation.
We quantitatively evaluate
BEVNet on both on-road and off-road scenarios, and show that it outperforms avariety of strong baselines.
Introduction
While there has been great recent interest in the development of autonomous vehicles, the vastmajority of this work has focused on on-road and urban driving.
However, a wide range of application areas including defence, agriculture, conservation, and search and rescue could beneﬁt fromautonomous off-road vehicles that can operate in complex, natural terrain.
In such environments,understanding the traversability of terrain surrounding the vehicle is crucial for successful planningand control.
Perceiving whether terrain is actually traversable from sparse LiDAR data can be achallenging problem as off-road terrain is often characterized by rapid changes to the ground plane,heavy vegetation, overhanging branches, and negative obstacles.
In other words, a successful offroad robot must reason about both the geometric and semantic content of its surroundings in orderto determine what terrain is traversable and what is impassible.
An effective terrain traversability prediction system should efﬁciently 1) aggregate the observationsover time [1, 2] with noisy odometry [3, 4], 2) reason about the partially seen or even yet to be seenparts of the environment [5, 6, 7], and 3) detect overhanging structures in the environment, suchas tree branches, tunnels, and power-lines [8, 9].
While previous work has addressed each of theseissues individually, the aforementioned challenges are related, and solving each one should beneﬁtthe others.
In this paper, we propose a Bird’s Eye View Network (BEVNet), a recurrent neural network thatdirectly predicts the traversability of the terrain in the form of a 2D grid around the robot from
LiDAR scans.
As shown in Figure 1, our model has three main parts: 1) a 3D sparse convolution subnetwork to process the voxelized point cloud, 2) a Convolutional Gated Recurrent Unit (ConvGRU)which uses convolutional layers in gated recurrent unit [10] to aggregate the 3D information, 3)a 2D convolutional encoder-decoder with efﬁcient backbone [11] that simultaneously inpaints theempty spaces and projects the 3D data into the 2D Bird’s Eye View (BEV) map.
To train the model,we use both past and future labeled LiDAR scans to build a complete 3D semantic point cloud andbuild the ground-truth 2D traversability map.
Unlike previous work [8], our model does not use anyexplicit rule to ﬁnd the overhanging structures when doing the projection.
We detect and removeany overhanging structures when building the ground-truth BEV map which can be done efﬁciently
[Figure]
as the environment is fully observed and labeled.
The network learns what should be considered asa overhanging obstacle given sparse LiDAR scans.
We make several contributions and empirical observations.
We proposed a novel framework to buildthe BEV map by simultaneously 1) aggregating observations over time, 2) predicting the unseenareas of the map, and 3) ﬁltering out irrelevant obstacles like overhanging tree branches that do notaffect traversability.
Experimental results on SemanticKITTI [12] and RELLIS-3D [13] demonstratethe novel framework works well in both on-road and off-road settings, and outperforms the strongbaselines in all the aforementioned tasks.
Related Work
While there exists a vast literature on perception for autonomous driving [14, 15, 16, 17, 18, 19,20, 21], most prior work focuses on urban environments and leverage large datasets and structureinherent in cities and road networks.
Since there are not as many recent papers focused on offroad driving and rural environments, we compare our system to prior works that share the mutualcomponents to ours, namely traversability analysis [22, 23, 24, 25], online semantic mapping [15,16, 1, 17, 21], recurrency handling [15, 1, 17], bird’s eye view semantic segmentation [1, 19, 26, 27],and semantic scene completion [5, 6, 7].
Traversability Analysis Traversability Analysis is less common in the task of urban autonomousdriving but crucial for a successful off-road autonomy [23, 28, 29].
Traversability of surroundingterrain may be analyzed based on various criteria, including surface roughness [22], negative obstacles [23], and terrain classiﬁcation [24, 25].
Our system performs end-to-end semantic mapping byclassifying each surrounding location based on traversabilty.
Online Semantic Mapping Semantic mapping provides structured information for an autonomous driving system, with the earliest works in this area dating back to well before deeplearning driven methods gained traction [30].
A core paradigm is the bird’s eye view representation,which stores local information in a two-dimensional grid surrounding the vehicle [31].
While priorworks such as [14, 15, 16] utilize a high deﬁnition map as additional prior knowledge, such a mapmay not be available for off-road terrain.
Our system therefore does not assume a map a priori , andinstead constructs one online.
Similar to our system, Casas et al.
[17], Wu et al.
[21] learns to produce semantic maps for urban autonomous driving.
In comparison, our system focuses on semanticmapping with a broader categorization of semantics based on terrain traversability, which is usefulfor off-road driving.
Recurrent Representations Temporally consistent accumulation of semantic information is crucial for mapping the environment and providing ture is speciﬁcally designed for semantic occupancy forecasting, whereas our recurrent architectureaccumulates sensor data to better estimate the traversability of the current surrounding terrain.
Maturana et al.
[1] utilize a more conventional approach by accumulating information via Bayes ﬁltering,which in our system is replaced with a recurrent neural network.
There are various approaches tohandling recurrency in neural networks including [2, 32, 33, 34, 35, 36, 37].
Our system utilizes
ConvGRU to accumulate 2D representations for BEV mapping.
BEV Semantic Segmentation Philion and Fidler [19] learns to produce a BEV map from RGBcamera for on-road driving, and Maturana et al.
[1] produces a BEV map for off-road driving bysegmenting an image and projecting it using depth information.
In comparison, our system learnsto directly project a LiDAR scan to produce a BEV map.
We also compare our approach to LiDARsegmentation where the segmented point cloud from networks such as [26, 27] can be projected ontoa BEV map.
We evaluate this comparison in detail in Sec. 5.
Semantic Scene Completion (SSC)
The goal of SSC is to generate a complete 3D scene givena single LiDAR scan as input.
Existing works such as [5, 6, 7] utilize information from semanticsegmentation to complete the scene, whereas our system learns to directly predict the completedscene and therefore does not require segmentation prediction from a secondary network or groundtruth labels.
In addition, our system performs point cloud projection and scene completion in 2Dsimultaneously, as the main task in our work is to produce a 2D BEV map.
We evaluate our system’sability to complete scenes by comparing our model against [5], the details can be found in Sec. 5.
Method
3.1 Overview
We consider a mobile robot with a 360 ◦ LiDAR mounted at its top.
In order for the robot to navigateefﬁciently and safely in a new environment (either on-road or off-road), the robot builds an onlinetraversability map around itself.
The traversability map resembles a conventional occupancy map aswell as the semantic map from [1], where each cell stores a probabilitity distribution of traversabilitylabels.
In this work, we use four levels of traversability: free , low-cost , medium-cost and lethal .
Thenumber of traversability levels can be trivially extended if so desired.
The traversability map isinside the robot’s odometry frame, so that the robot is always at the center, with its heading pointingto the east.
The traversability map is converted to a costmap by mapping each traversability levelto the corresponding cost value via a lookup table.
The converted costmap can be easily interfacedwith a local planner [38] or a global planner (e.g., A*) for ﬁnding the least-cost path to a goal.
We adopt a supervised-learning approach to predict this traversability map.
We start by buildinga traversability dataset from LiDAR segmentation datasets [12, 13] via a traversability-aware projection procedure.
Then, we introduce BEVNet, a recurrent neural network that takes the current
LiDAR scan and utilizes its history to build a dense traversability map.
In the following sections,we will describe each component in detail.
3.2 Building a Traversability Dataset
Recent work [15, 1] focuses on on-road driving where reasoning about a large number of ﬁnegrained semantic classes is necessary.
Here we consider a more general driving paradigm where wesimply care about the traversability of the surrounding terrain.
This makes our model applicableto both on-road and off-road driving.
Given a dataset with semantically labeled LiDAR scans, weconvert it to a traversability dataset via the following procedure (illustrated in Figure 2).
Scan Aggregation.
For each scan, we aggregate it with the past t and the future t scans with strides to construct a larger point set.
We set t to a large enough number (e.g., 71) to obtain densetraversability information for a large area around the robot.
These parameters may be tuned depending on the vehicle speed and density of the LiDAR points.
Traversability Mapping.
We map the semantic classes into our 4-level ontology.
The general principle is to map semantic classes with similar costs to the same traversability label.
For example,car and building are mapped to lethal , whereas mud and grass are mapped to low-cost .
Detailedmapping can be found in the supplementary.
[Figure]
Points Binning and Ground Height Estimation.
For each point in the aggregated scan, we do adown projection to ﬁnd its location x, y on the traversability map.
Hence each x, y location of themap contains a pillar of points.
We estimate the ground height map by running a mean ﬁlter kernelover the lowest z coordinates of the points labeled as free and low-cost at each x, y location in themap.
This height map is used as a reference for ﬁnal traversability projection.
Traversability Projection.
For each pillar of points, we ﬁlter out overhanging obstacles by removing points that are above the local ground level by a certain threshold because they will not collidewith the robot.
For the remaining points, we take the point with the lowest traversability at each x, ylocation as the ﬁnal traversability label.
3.3 Feature Extraction via Sparse Convolution with Z Compression
The architecture of BEVNet is shown in Figure 1.
An input LiDAR scan is ﬁrst discretized into a512 × 512 × 31 grid with a resolution of 0 .
2 m .
We perform sparse discretization so that only occupiedvoxels are preserved.
Each voxel contains a 4-dimensional feature f = 1 n (cid:80) ni
=
[ x i , y i , z i , r i ] , whichis the average of the coordinates and remission values of the points inside the voxel.
This sparsevoxel grid is fed into a sequence of sparse convolution layers, which compress the z dimension viastrided convolutions.
We keep x and y dimensions unchanged.
The output of the sparse convolutionlayers is a sparse feature tensor S of size 512 × 512 × C , where C is the feature dimension.
3.4 Temporal Aggregation of Sparse Feature Maps
A single LiDAR scan becomes increasingly sparse as the distance increases, making it difﬁcult to estimate traversability for areas far away from the robot.
Contrary to classical SLAM that aggregates
LiDAR measurements over time via a hand-engineered Bayesian update rule [39], we let the network learn to aggregate the sparse feature maps from past LiDAR scans via a Convolutional Gated
Recurrent Unit (ConvGRU).
The ConvGRU maintains a 2D latent feature map M that shares the same coordinate system and dimensions as the ﬁnal traversability map.
The latent feature map M is updated as
[Equation]
where ∆ T t +1 is the relative transform of the robot’s odometry frame from t to t +1 .
The WarpAfﬁneoperation transforms the latent feature map M t from the previous odometry frame to the current odometry frame so that the features from M t and S t +1 are spatially aligned.
Note that the
WarpAfﬁne operation is differentiable to allow the gradients to backpropagate through time.
3.5 Traversability Inpainting
Since the ConvGRU only aggregates sparse feature tensors, M contains little information for areaswhere there is no LiDAR point.
Instead of treating no-hit area as unknown, we let the network ﬁll inthe empty space by leveraging the local and global contextual cues via the Inpainting Network .
Theinpainting network is a fully convolutional network inspired by FCHardNet [11] which is originallydesigned for fast image segmentation.
It consists of a sequence of downsampling and upsamplinglayers with skip connections, making it effective for capturing local and global contextual information for predicting what is missing.
Implementation Details
We build the traversability datasets from SemanticKITTI [12] and RELLIS-3D [13] to evaluate
BEVNet in both on-road and off-road scenarios.
For SemanticKITTI we aggregate 71 frames withstride 2 to generate a single traversability map.
For RELLIS-3D we aggregate 141 frames with stride5.
Both datasets provide per-frame odometry, which we use for the differential warping layer in the
ConvGRU.
The traversability maps have a size of 102 .
4 m × 102 .
4 m with a resolution of 0 .
2 m .
Notethat the traversability maps contain an additional “unknown” class marking regions that have neverbeen observed.
Network training.
We train our network using the Adam optimizer [40] with an initial learningrate of 3 e − 4 and a decay of 0.7 per epoch.
We use a weighted Cross-Entropy loss.
We startwith training a single-frame model without ConvGRU until the model converges.
Then we freezethe sparse convolution layers and insert the ConvGRU layer, and then train the ConvGRU and theinpainting network together.
While technically we can train the whole network end-to-end , thistwo-stage training procedure is faster and is more memory-efﬁcient.
When training the ConvGRU,we use a sequence length of 5 with a frame stride randomly chosen from [1, 10, 20].
Training takesabout 12 hours on a single RTX 3090.
The inference time of our network is 6 fps on a RTX 3090.
Data augmentation.
During training, we randomly rotate every pair of LiDAR scans and theground truth traversability map in U [ − 45 ◦ , 45 ◦ ] , and randomly drop 20% of the points.
Furthermore,we perturb the groundtruth odometry with rotation drawn from N (0 , 0 .
and translation drawnfrom N (0 , 0 .
Note that the error in odometry will accumulate over time.
We evaluate the effectof noisy odometry in Sec. 5.
Experiments
We conduct both quantitative and qualitative study on SemanticKITTI (on-road) and RELLIS-3D(off-road) datasets.
We trained a separate model for each dataset.
We compare with a variety ofbaselines, ranging from LiDAR segmentation to scene completion on the validation sequences.
Wealso perform an ablation study to better understand the contribution of recurrence, and how ourmodel behaves on the two datasets that have very different characteristics.
5.1 Evaluation Metrics
We use the mean Intersection of Union (mIoU) [11], a widely used metric for image segmentation,as the quantitative measure of the prediction accuracy.
Note that our model predicts an additional“unknown” class to improve the visual consistency, and we exclude the “unknown” class in theevaluation.
To better understand our model’s capability of predicting the future, we report mIoUsin three modes: seen , unseen , and all .
In the “seen” mode, we do not include ground truth labelsobtained from future frames, effectively excluding any future predictions.
For the “unseen” modelwe only include the future predictions.
In the “all” scenario, we evaluate on both.
5.2 Comparison with LiDAR Segmentation with Temporal Aggregation
A strong baseline for building a traversability map is to perform semantic segmentation of the incoming LiDAR scan, project it down to obtain a 2D sparse traversability map, and aggregate the
[Table]
traversability maps over time.
To compare with this approach, we choose Cylinder3D [26] (ﬁnetuned on our 4-class ontology) as the LiDAR segmentation network for its strong performance, anduse the same projection procedure in Sec 3 on the input
LiDAR scan to obtain the single-frametraversability map.
We perform the temporal aggregation by tracking the categorical distribution oftraversability via a uniform Dirichlet prior.
To do so, we keep a counter map M C of size H × W × 4(initialized to zeros).
It is of the same size as the traversability map except that the last dimensioncounts the traversability labels observed so far.
We update M C incrementally.
For each incomingsingle-frame traversability map, we warp M C to the current odometry frame via bilinear interpolation, and increment the counts by adding the one-hot version of the incoming single-frame map.
The actual traversability label can be obtained by taking the argmax of the last dimension of M C .
Results on SemanticKITTI.
In the left half of Table 1 we compare the performance of BEVNetRecurrent (BEVNet-R) with Cylinder3D+Temporal Aggregation (C3D-TA) on the SemanticKITTIvalidation set.
When only considering what has been observed so far (“seen”) and clean odometry,
C3D-TA is better than
BEVNet-R.
This shows that LiDAR segmentation with accurate temporalaggregation can work very well in structured environments such as on-road driving.
When evaluatedon the full groundtruth (“full”), BEVNet-R outperforms C3D-TA because C3D-TA cannot predictthe future traversability.
When evaluating on noisy odometry, BEVNet-R surpasses C3D-TA for both“seen” and “full” test scenarios.
BEVNet-R uses learned recurrency to “ﬁx” the errors in odometryand to adaptively forget history in case the error in odometry is too large.
In comparison, C3D-TAsolely uses the provided odometry to aggregate information, which may result in large misalignmentas errors accumulate over time.
Results on RELLIS-3D.
The results on RELLIS-3D (right half of Table 1) share a similar trendas those in SemanticKITTI, except that BEVNet-R consistently outperforms C3D-TA with a largergap.
This suggests that off-road environment is more challenging, where accurate LiDAR segmentation is hard to obtain due to the lack of environmental structure.
Indeed, Cylinder3D only achievesa 64.1 mIoU on RELLIS-3D for LiDAR segmentation, which is lower than the 87.9 mIoU on SemanticKITTI.
Interestingly, noisy odometry has almost no impact on BEVNet-R.
We hypothesizethat it is because the RELLIS-3D dataset contains less clutter and occlusion so BEVNet-R does notrely heavily on the history for traversability prediction.
Qualitative results In left half of Figure 3, we highlight the fact that BEVNet-R can preserve smalldynamic objects such as bicyclists better than C3D-TA.
Hand-engineered temporal aggregation isprone to treating small dynamics objects as noise and ignoring them.
In comparison, BEVNet canlearn to keep small dynamic objects, while preserving smoothness in static regions.
The right halfshows the impact of noisy odometry.
We can see large misalignment and smear artefact for C3D-TA,whereas BEVNet-R produces signiﬁcantly cleaner output.
Finally, in Figure 4 we visualize examples on both SemanticKITTI and RELLIS-3D. In general BEVNet-R shows strong performance inpredicting future traversability.
It learns to predict whole cars, alley entrances, and trail paths withextremely sparse LiDAR points.
5.2.1 Ablation Study
We conduct our ablation study on three variants of BEVNet: BEVNet-Single (BEVNet-S), BEVNetSingle+Temporal Aggregation (BEVNet-TA), and BEVNet-Recurrent (BEVNet-R).
We aim to answer three questions: 1) is learned recurrence better than temporal aggregation?
2) does history help
[Figure]
predict the future?
and 3) where should information be aggregated in the network?
We answer thesequestions through a set of experiments on both SemanticKITTI and RELLIS-3D datasets.
Is learned recurrence better than temporal aggregation?
When evaluated on the full groundtruth, we observe that BEVNet-R consistently outperforms BEVNet and BEVNet-TA on bothon-road and off-road scenarios (Table 1).
Notably, BEVNet-TA also outperforms BEVNet, whichshows that any form of recurrence is beneﬁcial.
In particular, we observe that the learned recurrencemakes best use of the temporal information in comparison to the hand-engineered TA.
When noisyodometry is introduced we observe the same trend as discussed in Sec.
5.2, where BEVNet-R showsrobustness to noise and outperforms BEVNet-TA.
Does history help predict the future?
In Table 1, we can see that any form of recurrence thataccumulates history helps with predicting the unseen area.
When evaluated on unseen groundtruth, BEVNet-TA and BEVNet-R both consistently outperform BEVNet-S, even when noisyodometry is introduced.
This yields an interesting conclusion that any form of memory acquiredthrough recurrence provides more information for forecasting the future.
We emphasize that learnedrecurrence especially proves strong improvement in this aspect, as it consistently outperforms allother approaches on the unseen ground truth.
mIoU
Early Aggregation 0.
Late Aggregation 0.
Where to put ConvGRU?
Recurrence may be applied right afterthe sparse convolutions ( early aggregation ) or may be applied after the 2D inpainting network ( late aggregation ).
We compare thetwo approaches on the SemanticKITTI dataset with clean odometry and including the unseen area for evaluation.
Note that hereour model is trained with clean odometry.
As we emphasize with
Table 2, our experiments show that early aggregation yields better results than late aggregation.
Thisis because when early aggregation is applied the inpainting network has access to temporally fusedinformation, and therefore is given more information to complete the scene and maintain temporalconsistency across scans.
Furthermore, we may infer that if late aggregation is applied, it is moredifﬁcult for the recurrent network to learn to correct the odometry as it is given completed sceneswith potentially noisy information instead of the sparse feature maps.
5.3 Comparison and Discussion with other Related Work
Several recent papers have focused on semantic understanding of scenes from sparse LiDAR scans,which we discuss below.
These papers solve related, but slightly different problems.
They producesmaller maps, and do not perform temporal aggregation.
Semantic Scene Completion.
The scene completion task aims to predict a dense semantic voxelgrid from a single LiDAR scan.
To compare to our work, we build a traversability map by convertingpredicted dense voxel grids to 2D traversability maps using the same method described in Sec. 3.
We compare with JS3C-Net [5] because it is one of the top performing models on the SemanticKITTIscene completion task and the code is publicly available.
We map the 19-class predictions of JS3CNet to our 4-class ontology, and project the voxels down to generate 256 × 256 traversability maps.
We adapt our approach to produce the same output format as JS3C-Net.
JS3C-Net achieves an mIoUof 0.549, whereas BEVNet-S and BEVNet-R achieve an mIoU of 0.592 and 0.608, respectively.
Thisshows that learning to project traversability and inpainting the map simultaneously can work betterthan
ﬁrst reconstructing the scene, followed by a rule-based projection.
[Figure]
SemanticKITTI
SemanticKITTI SemanticKITTI
[Figure]
Inpainting Network.
Han et al.
[7] recently proposed an approach that uses GANs to inpaint asparsely segmented BEV image.
We trained our model using the same SemanticKITTI dataset withthe groundtruth 19-class BEV images as supervision.
Note that this task is different from traversability estimation because it does a simple topdown projection.
Our single-frame model achieves 0.253mIoU on this task, which is signiﬁcantly higher than 0.131 reported in [7].
Note that [7] assumesthe LiDAR scan has been already segmented, and it does inpainting in 2D.
This makes it unsuitablefor our traversability projection due to the lack of 3D reasoning.
Conclusion
We propose BEVNet, a framework that predicts dense terrain traversability in a local region arounda mobile robot with the aim of helping the robot navigate in a novel on-road or off-road environment.
BEVNet addresses a number of challenges in a uniﬁed architecture, namely: 1) it learns to aggregatesparse LiDAR information over time, 2) it learns to reason about traversability that involves both geometric and semantic understanding of the environment, and 3) it learns to ﬁll in the unknown spacewhere there are no LiDAR hits, and thus provides the robot with a more complete understanding ofits surroundings.
Most notably, BEVNet can leverage past information to better predict the future .
We believe BEVNet provides an important step towards robot autonomy on complex terrains wherea prior map is unavailable.
In the future, we would like to test BEVNet in more challenging off-roadscenarios and predict traversability from cheap, weakly