Semantic Terrain Classiﬁcation for Off-Road Autonomous Driving
Anonymous Author(s)
AfﬁliationAddress email
Abstract: Producing dense and accurate traversability maps is crucial for autonomous off-road navigation. In this paper, we focus on the problem of classifying terrains into 4 cost classes (free, low-cost, medium-cost, obstacle) fortraversability assessment.
This requires a robot to reason about both semantics (what objects are present?) and geometric properties (where are the objects located?) of the environment.
To achieve this goal, we develop a novel Bird’s Eye
View Network (BEVNet), a deep neural network that directly predicts a local mapencoding terrain classes from sparse LiDAR inputs.
BEVNet processes both geometric and semantic information in a temporally consistent fashion.
More impor- tantly, it uses learned prior and history to predict terrain classes in unseen spaceand into the future, allowing a robot to better appraise its situation.
We quantitatively evaluate BEVNet on both on-road and off-road scenarios, and show that itoutperforms a variety of strong baselines.
Introduction
While there has been great recent interest in the development of autonomous vehicles, the vastmajority of this work has focused on on-road and urban driving.
However, a wide range of application areas including defence, agriculture, conservation, and search and rescue could beneﬁt fromautonomous off-road vehicles that can operate in complex, natural terrain.
In such environments,understanding the traversability of terrain surrounding the vehicle is crucial for successful planningand control.
Perceiving whether terrain is actually traversable from sparse LiDAR data can be achallenging problem as off-road terrain is often characterized by rapid changes to the ground plane,heavy vegetation, overhanging branches and negative obstacles.
In other words, a successful offroad robot must reason about both the geometric and semantic content of its surroundings in orderto determine what terrain is traversable and what is impassible.
In this work we formulate traversability estimation as a semantic terrain classiﬁcation problem [1,2].
The motivation is to unify the semantics (what objects are present?)
and geometry (where are theobjects located?)
of the terrain into a single cost ontology.
From the semantic perspective, objectssuch as large rocks and tree trunks are non-traversable, whereas gravel road, grassy area, and bushesare all traversable by an off-road vehicle [3] but with increasing difﬁculty. From the geometricperspective, overhanging obstacles can be ignored, and objects of the same semantic class may varyin their traversability depending on their heights (e.g., tall bushes vs. short bushes). To this end,we use a discrete set of traversability levels for the ease of grouping the semantic classes by theirtraversability while allowing us to adjust the traversability levels of speciﬁc instances based on theirgeometry. An effective terrain classiﬁcation system should efﬁciently 1) aggregate observations over time [1,4] with noisy odometry [5, 6], 2) reason about the partially seen or even yet to be seen parts ofthe environment [7, 8, 9], and 3) detect overhanging structures such as tree branches, tunnels, andpower-lines [10, 11].
While previous work has addressed each of these issues individually, theaforementioned challenges are related, and solving each one should beneﬁt the others.
[Figure] In this paper, we propose a Bird’s Eye View Network (BEVNet), a recurrent neural network thatdirectly predicts terrain classes in the form of a 2D grid around the robot from LiDAR scans.
Asshown in Figure 1, our model has three main parts: 1) a 3D sparse convolution sub-network toprocess the voxelized point cloud, 2) a Convolutional Gated Recurrent Unit (ConvGRU) which usesconvolutional layers in a gated recurrent unit [12] to aggregate the 3D information, 3) an efﬁcient2D convolutional encoder-decoder based on [13] that simultaneously inpaints the empty spacesand projects the 3D data into the 2D Bird’s Eye View (BEV) map.
To train the model, we useboth past and future labeled LiDAR scans to build a complete 3D semantic point cloud and buildthe ground-truth 2D traversability map.
Unlike previous work [10], our model does not use anyexplicit rule to ﬁnd the overhanging structures when doing the projection.
We detect and remove anyoverhanging structures when building the ground-truth BEV costmap which can be done efﬁcientlyas the environment is fully observed and labeled.
The network learns what should be considered asa overhanging obstacle given sparse LiDAR scans.
We make several contributions and empirical observations.
We proposed a novel framework to buildthe BEV costmap by simultaneously 1) aggregating observations over time, 2) predicting the unseenareas of the map, and 3) ﬁltering out irrelevant obstacles like overhanging tree branches that do notaffect traversability.
Experimental results on SemanticKITTI [14] and RELLIS-3D [15] demonstratethe novel framework works well in both on-road and off-road settings, and outperforms the strongbaselines in all the aforementioned tasks.
Related Work
While there exists a vast literature on perception for autonomous driving [16, 17, 18, 19, 20, 21,22, 23], most prior work focuses on urban environments and leverage large datasets and structureinherent in cities and road networks.
Since there are not as many recent papers focused on offroad driving and rural environments, we compare our system to prior works that share the mutualcomponents to ours, namely traversability analysis [24, 25, 26, 27], online semantic mapping [17,18, 1, 19, 23], recurrency handling [17, 1, 19], bird’s eye view semantic segmentation [1, 21, 28, 29],and semantic scene completion [7, 8, 9].
Traversability Analysis Traversability Analysis is less common in the task of urban autonomousdriving but crucial for a successful off-road autonomy [25, 30, 31].
Traversability of surroundingterrain may be analyzed based on various criteria, including surface roughness [24], negative obstacles [25], and terrain classiﬁcation [26, 27].
Our system performs end-to-end semantic mapping byclassifying each surrounding location based on traversabilty.
Online Semantic Mapping Semantic mapping provides structured information for an autonomous driving system, with the earliest works in this area dating back to well before deeplearning driven methods gained traction [32].
A core paradigm is the bird’s eye view representation,which stores local information in a two-dimensional grid surrounding the vehicle [33].
While prior works such as [16, 17, 18] utilize a high deﬁnition map as additional prior knowledge, such a mapmay not be available for off-road terrain.
Our system therefore does not assume a map a priori , andinstead constructs one online.
Similar to our system, Casas et al.
[19], Wu et al.
[23] learns to produce semantic maps for urban autonomous driving.
In comparison, our system focuses on semanticmapping with a broader categorization of semantics based on terrain traversability, which is usefulfor off-road driving.
Recurrent Representations Temporally consistent accumulation of semantic information is crucial for mapping the environment and providing the information necessary for safe, efﬁcient motionplanning. Recent work [17, 19] directly concatenates the past 10 voxelized LiDAR scans as its inputrepresentation for memory efﬁciency. In comparison to our system, the author’s recurrent architecture is speciﬁcally designed for semantic occupancy forecasting, whereas our recurrent architectureaccumulates sensor data to better estimate the traversability of the current surrounding terrain.
Maturana et al.
[1] utilize a more conventional approach by accumulating information via Bayes ﬁltering,which in our system is replaced with a recurrent neural network.
There are various approaches tohandling recurrency in neural networks including [4, 34, 35, 36, 37, 38, 39].
Our system utilizes
ConvGRU to accumulate 2D representations for BEV mapping.
BEV Semantic Segmentation Philion and Fidler [21] learns to produce a BEV map from RGBcamera for on-road driving, and Maturana et al.
[1] produces a BEV map for off-road driving bysegmenting an image and projecting it using depth information.
In comparison, our system learnsto directly project a LiDAR scan to produce a BEV map.
We also compare our approach to LiDARsegmentation where the segmented point cloud from networks such as [28, 29] can be projected ontoa BEV map.
We evaluate this comparison in detail in Sec. 5.
Semantic Scene Completion (SSC)
The goal of SSC is to generate a complete 3D scene givena single LiDAR scan as input.
Existing works such as [7, 8, 9] utilize information from semanticsegmentation to complete the scene, whereas our system learns to directly predict the completedscene and therefore does not require segmentation prediction from a secondary network or groundtruth labels.
In addition, our system performs point cloud projection and scene completion in 2Dsimultaneously, as the main task in our work is to produce a 2D BEV map.
We evaluate our system’sability to complete scenes by comparing our model against [7], the details can be found in Sec. 5.
Method
3.1 Overview
We consider a mobile robot with a 360 ◦ LiDAR mounted at its top.
In order for the robot to navigateefﬁciently and safely in a new environment (either on-road or off-road), the robot builds an onlinetraversability map around itself.
The traversability map resembles a conventional occupancy map aswell as the semantic map from [1], where each cell stores a probability distribution of traversabilitylabels.
In this work, we use four levels of traversability: free , low-cost , medium-cost and lethal .
The number of traversability levels can be trivially extended if so desired.
The traversability map isinside the robot’s odometry frame, so that the robot is always at the center, with its heading pointingto the east.
The traversability map is converted to a costmap by mapping each traversability levelto the corresponding cost value via a lookup table.
The converted costmap can be easily interfacedwith a local planner [40] or a global planner (e.g., A*) for ﬁnding the least-cost path to a goal.
We adopt a supervised-learning approach to predict this traversability map.
We start by buildinga traversability dataset from LiDAR segmentation datasets [14, 15] via a traversability-aware projection procedure.
Then, we introduce BEVNet, a recurrent neural network that takes the current
LiDAR scan and utilizes its history to build a dense traversability map.
In the following sections,we will describe each component in detail.
3.2 Building a Traversability Dataset
Recent work [17, 1] focuses on on-road driving where reasoning about a large number of ﬁnegrained semantic classes is necessary.
Here we consider a more general driving paradigm where wesimply care about the traversability of the surrounding terrain.
This makes our model applicable [Figure] to both on-road and off-road driving.
Given a dataset with semantically labeled LiDAR scans, weconvert it to a traversability dataset via the following procedure (illustrated in Figure 2).
Scan Aggregation.
For each scan, we aggregate it with the past t and the future t scans with strides to construct a larger point set.
We set t to a large enough number (e.g., 71) to obtain densetraversability information for a large area around the robot.
These parameters may be tuned depending on the vehicle speed and density of the LiDAR points.
Traversability Mapping.
We map the semantic classes into our 4-level ontology.
The general principle is to map semantic classes with similar costs to the same traversability label.
For example,car and building are mapped to lethal , whereas mud and grass are mapped to low-cost .
Detailedmapping can be found in the supplementary.
Points Binning and Ground Height Estimation.
For each point in the aggregated scan, we do adown projection to ﬁnd its location x, y on the traversability map.
Hence each x, y location of themap contains a pillar of points.
We estimate the ground height map by running a mean ﬁlter kernelover the lowest z coordinates of the points labeled as free and low-cost at each x, y location in themap.
This height map is used as a reference for ﬁnal traversability projection.
Traversability Projection.
For each pillar of points, we ﬁlter out overhanging obstacles by removing points that are above the local ground level by a certain threshold because they will not collidewith the robot.
Additionally, we adjust the traversability level of certain points based on their heightabove the ground level and the mobile capabilities of the robot.
i.e. points labelled as mediumcost but very close to the local ground level can be deemed negligible for a large off-road vehicleand hence remapped to low-cost as other nearby points.
Finally, we take the point with the lowesttraversability (i.e., most difﬁcult) at each x, y location as the ﬁnal traversability label.
3.3 Feature Extraction via Sparse Convolution with Z Compression
The architecture of BEVNet is shown in Figure 1.
An input LiDAR scan is ﬁrst discretized into a512 × 512 × 31 grid with a resolution of 0 .
2 m .
We perform sparse discretization so that only occupiedvoxels are preserved.
Each voxel contains a 4-dimensional feature f = 1 n (cid:80) ni
=
[ x i , y i , z i , r i ] , whichis the average of the coordinates and remission values of the points inside the voxel.
This sparsevoxel grid is fed into a sequence of sparse convolution layers, which compress the z dimension viastrided convolutions.
We keep x and y dimensions unchanged.
The output of the sparse convolutionlayers is a sparse feature tensor S of size 512 × 512 × C , where C is the feature dimension.
3.4 Temporal Aggregation of Sparse Feature Maps
A single LiDAR scan becomes increasingly sparse as the distance increases, making it difﬁcult to classify the traversability level for areas far away from the robot.
Contrary to classical SLAM that aggregates LiDAR measurements over time via a hand-engineered Bayesian update rule [41], we let the network learn to aggregate the sparse feature maps from past LiDAR scans via a Convolutional
Gated Recurrent Unit (ConvGRU).
The ConvGRU maintains a 2D latent feature map M that shares the same coordinate system and dimensions as the ﬁnal traversability map.
The latent feature map M is updated as
[Equation]
where ∆ T t +1 is the relative transform of the robot’s odometry frame from t to t +1 .
The WarpAfﬁneoperation transforms the latent feature map M t from the previous odometry frame to the current odometry frame so that the features from M t and S t +1 are spatially aligned.
Note that the
WarpAfﬁne operation is differentiable to allow the gradients to backpropagate through time.
3.5 Traversability Inpainting
Since the ConvGRU only aggregates sparse feature tensors, M contains little information for areaswhere there is no LiDAR point.
Instead of treating no-hit area as unknown, we let the network ﬁll inthe empty space by leveraging the local and global contextual cues via the Inpainting Network .
Theinpainting network is a fully convolutional network inspired by FCHardNet [13] which is originallydesigned for fast image segmentation.
It consists of a sequence of downsampling and upsamplinglayers with skip connections, making it effective for capturing local and global contextual information for predicting what is missing.
Implementation Details
We build the traversability datasets from SemanticKITTI [14] and RELLIS-3D [15] to evaluate
BEVNet in both on-road and off-road scenarios.
For SemanticKITTI we aggregate 71 frames withstride 2 to generate a single traversability map.
For RELLIS-3D we aggregate 141 frames with stride5.
Both datasets provide per-frame odometry, which we use for the differential warping layer in the
ConvGRU.
The traversability maps have a size of 102 .
4 m × 102 .
4 m with a resolution of 0 .
2 m .
Note that the traversability maps contain an additional “unknown” class marking regions that havenever been observed.
For additional details on network training and data augmentation please referto section 4 in the supplementary material.
Experiments
We conduct both quantitative and qualitative study on SemanticKITTI (on-road) and RELLIS-3D(off-road) datasets.
We trained a separate model for each dataset.
We compare with a variety ofbaselines, ranging from LiDAR segmentation to scene completion on the validation sequences.
Wealso perform an ablation study to better understand the contribution of recurrence, and how ourmodel behaves on the two datasets that have very different characteristics.
5.1 Evaluation Metrics
We use the mean Intersection of Union (mIoU) [13], a widely used metric for image segmentation,as the quantitative measure of the prediction accuracy.
Note that our model predicts an additional“unknown” class to improve the visual consistency, and we exclude the “unknown” class in theevaluation.
To better understand our model’s capability of predicting the future, we report mIoUsin three modes: seen , unseen , and all .
In the “seen” mode, we do not include ground truth labelsobtained from future frames, effectively excluding any future predictions.
For the “unseen” modelwe only include the future predictions.
In the “all” scenario, we evaluate on both.
5.2 Comparison with LiDAR Segmentation with Temporal Aggregation
A strong baseline for building a traversability map is to perform semantic segmentation of the incoming LiDAR scan, project it down to obtain a 2D sparse traversability map, and aggregate thetraversability maps over time.
To compare with this approach, we choose Cylinder3D [28] (ﬁnetuned on our 4-class ontology) as the LiDAR segmentation network for its strong performance, anduse the same projection procedure in Sec 3 on the input
LiDAR scan to obtain the single-frametraversability map.
We perform the temporal aggregation by tracking the categorical distribution of [Table] traversability via a uniform Dirichlet prior.
To do so, we keep a counter map M C of size H × W × 4(initialized to zeros).
It is of the same size as the traversability map except that the last dimensioncounts the traversability labels observed so far.
We update M C incrementally.
For each incomingsingle-frame traversability map, we warp M C to the current odometry frame via bilinear interpolation, and increment the counts by adding the one-hot version of the incoming single-frame map.
The actual traversability label can be obtained by taking the argmax of the last dimension of M C .
Results on SemanticKITTI.
In the left half of Table 1 we compare the performance of BEVNetRecurrent (BEVNet-R) with Cylinder3D+Temporal Aggregation (C3D-TA) on the SemanticKITTIvalidation set.
When only considering what has been observed so far (“seen”) and clean odometry,
C3D-TA is better than
BEVNet-R.
This shows that LiDAR segmentation with accurate temporalaggregation can work very well in structured environments such as on-road driving.
When evaluatedon the full groundtruth (“full”), BEVNet-R outperforms C3D-TA because C3D-TA cannot predictthe future traversability.
When evaluating on noisy odometry, BEVNet-R surpasses C3D-TA for both“seen” and “full” test scenarios.
BEVNet-R uses learned recurrency to “ﬁx” the errors in odometryand to adaptively forget history in case the error in odometry is too large.
In comparison, C3D-TAsolely uses the provided odometry to aggregate information, which may result in large misalignmentas errors accumulate over time.
Results on RELLIS-3D.
The results on RELLIS-3D (right half of Table 1) share a similar trendas those in SemanticKITTI, except that BEVNet-R consistently outperforms C3D-TA with a largergap.
This suggests that off-road environment is more challenging, where accurate LiDAR segmentation is hard to obtain due to the lack of environmental structure.
Indeed, Cylinder3D only achievesa 64.1 mIoU on RELLIS-3D for LiDAR segmentation, which is lower than the 87.9 mIoU on SemanticKITTI.
Interestingly, noisy odometry has almost no impact on BEVNet-R.
We hypothesizethat it is because the RELLIS-3D dataset contains less clutter and occlusion so BEVNet-R does notrely heavily on the history for traversability prediction.
Qualitative results In left half of Figure 3, we highlight the fact that BEVNet-R can preserve smalldynamic objects such as bicyclists better than C3D-TA.
Hand-engineered temporal aggregation isprone to treating small dynamics objects as noise and ignoring them.
In comparison, BEVNet canlearn to keep small dynamic objects, while preserving smoothness in static regions.
The right halfshows the impact of noisy odometry.
We can see large misalignment and smear artefact for C3D-TA,whereas BEVNet-R produces signiﬁcantly cleaner output.
Finally, in Figure 4 we visualize examples on both SemanticKITTI and RELLIS-3D. In general BEVNet-R shows strong performance inpredicting future traversability.
It learns to predict whole cars, alley entrances, and trail paths withextremely sparse LiDAR points.
5.2.1 Ablation Study
We conduct our ablation study on three variants of BEVNet: BEVNet-Single (BEVNet-S), BEVNetSingle+Temporal Aggregation (BEVNet-TA), and BEVNet-Recurrent (BEVNet-R).
We aim to answer three questions: 1) is learned recurrence better than temporal aggregation?
2) does history helppredict the future?
and 3) where should information be aggregated in the network?
We answer thesequestions through a set of experiments on both SemanticKITTI and RELLIS-3D datasets.
Is learned recurrence better than temporal aggregation?
When evaluated on the full groundtruth, we observe that BEVNet-R consistently outperforms BEVNet and BEVNet-TA on both [Figure] on-road and off-road scenarios (Table 1).
Notably, BEVNet-TA also outperforms BEVNet, whichshows that any form of recurrence is beneﬁcial.
In particular, we observe that the learned recurrencemakes best use of the temporal information in comparison to the hand-engineered TA.
When noisyodometry is introduced we observe the same trend as discussed in Sec.
5.2, where BEVNet-R showsrobustness to noise and outperforms BEVNet-TA.
Does history help predict the future?
In Table 1, we can see that any form of recurrence thataccumulates history helps with predicting the unseen area.
When evaluated on unseen ground truth, BEVNet-TA and BEVNet-R both consistently outperform BEVNet-S, even when noisyodometry is introduced.
This yields an interesting conclusion that any form of memory acquiredthrough recurrence provides more information for forecasting the future.
We emphasize that learnedrecurrence especially proves strong improvement in this aspect, as it consistently outperforms allother approaches on the unseen ground truth.
mIoU
Early Aggregation 0.
Late Aggregation 0.
Where to put ConvGRU?
Recurrence may be applied right afterthe sparse convolutions ( early aggregation ) or may be applied after the 2D inpainting network ( late aggregation ).
We compare thetwo approaches on the SemanticKITTI dataset with clean odometry and including the unseen area for evaluation.
Note that hereour model is trained with clean odometry.
As we emphasize with
Table 2, our experiments show that early aggregation yields better results than late aggregation.
Thisis because when early aggregation is applied the inpainting network has access to temporally fusedinformation, and therefore is given more information to complete the scene and maintain temporalconsistency across scans.
Furthermore, we may infer that if late aggregation is applied, it is moredifﬁcult for the recurrent network to learn to correct the odometry as it is given completed sceneswith potentially noisy information instead of the sparse feature maps.
5.3 Comparison and Discussion with other Related Work
Several recent papers have focused on semantic understanding of scenes from sparse LiDAR scans,which we discuss below.
These papers solve related, but slightly different problems.
They producesmaller maps, and do not perform temporal aggregation.
Semantic Scene Completion.
The scene completion task aims to predict a dense semantic voxelgrid from a single LiDAR scan.
To compare to our work, we build a traversability map by convertingpredicted dense voxel grids to 2D traversability maps using the same method described in Sec. 3.
We compare with JS3C-Net [7] because it is one of the top performing models on the SemanticKITTIscene completion task and the code is publicly available.
We map the 19-class predictions of JS3CNet to our 4-class ontology, and project the voxels down to generate 256 × 256 traversability maps.
We adapt our approach to produce the same output format as JS3C-Net.
JS3C-Net achieves an mIoUof 0.549, whereas BEVNet-S and BEVNet-R achieve an mIoU of 0.592 and 0.608, respectively.
Thisshows that learning to project traversability and inpainting the map simultaneously can work betterthan
ﬁrst reconstructing the scene, followed by a rule-based projection.
Inpainting Network.
Han et al.
[9] recently proposed an approach that uses GANs to inpaint asparsely segmented BEV image.
We trained our model using the same SemanticKITTI dataset withthe groundtruth 19-class BEV images as supervision.
Note that this task is different from traversability estimation because it does a simple topdown projection.
{+Our single-frame model achieves 0.
+}
[Figure]
SemanticKITTI SemanticKITTI [Figure] mIoU on this task, which is signiﬁcantly higher than 0.131 reported in [9].
Note that [9] assumesthe LiDAR scan has been already segmented, and it does inpainting in 2D.
This makes it unsuitablefor our traversability projection due to the lack of 3D reasoning.
Conclusion
We propose BEVNet, a framework that classiﬁes terrain traversability in a local region around amobile robot with the aim of helping the robot navigate in a novel on-road or off-road environment.
BEVNet addresses a number of challenges in a uniﬁed architecture, namely: 1) it learns to aggregatesparse LiDAR information over time, 2) it learns to reason about traversability that involves both geometric and semantic understanding of the environment, and 3) it learns to ﬁll in the unknown spacewhere there are no LiDAR hits, and thus provides the robot with a more complete understanding ofits surroundings.
Most notably, BEVNet can leverage past information to better predict the future .
We believe BEVNet provides an important step towards robot autonomy on complex terrains wherea prior map is unavailable.
Limitations and Future work.
Due to the fact that the publicly available datasets do not containrough terrains, we do not take the roughness of terrains into account at the moment. However, BEVNet can be extended to predict additional quantities such as an elevation map to be fused withthe current semantic costmap for more accurate traversability estimation. In the future, we wouldlike to extend the capability of BEVNet and test it in more challenging off-road scenarios.