Semi-Local Convolutions for LiDAR Scan Processing
Abstract
A number of applications, such as mobile robots or automated vehicles, use LiDAR sensors to obtain detailed information about their three-dimensional surroundings.
Many methods use image-like projections to efﬁciently process these LiDAR measurements and use deep convolutional neural networks to predict semantic classes for each point in the scan.
The spatial stationary assumption enables the usage of convolutions.
However, LiDAR scans exhibit large differences in appearance over the vertical axis.
Therefore, we propose semi local convolution (SLC), a convolution layer with reduced amount of weight-sharing along the vertical dimension.
We are ﬁrst to investigate the usage of such a layer independent of any other model changes.
Our experiments did not show any improvement over traditional convolution layers in terms of segmentation IoU or accuracy.
Introduction
For a wide variety of applications it is important to understand the semantic meaning of the three dimensional world.
LiDAR sensors provide a precise sampling of the environment and are often used for object detection and semantic segmentation.
Many state-of-the-art semantic segmentation approaches make use of traditional two-dimensional convolutional neural networks (CNNs) by projecting the point clouds into an image-like structure [9, 15].
In this paper we provide deeper insights into one of the methods developed in our previous work [14].
We investigate whether the spatial stationary assumption of convolutions is still applicable to inputs with varying statistical properties over parts of the data, such as projected LiDAR scans.
These data structures exhibit similar features as aligned images for which locally connected layers have been introduced [13].
Fig.
1 shows an example of such a projected LiDAR scan which has a high variance in depth over the vertical axis, but not the horizontal axis.
Further, the reﬂecivity measurements show inconsistencies over the vertical axis.
To address this effect, we introduce SLC, a layer with reduced weight-sharing along the vertical spatial dimension.
Related Work
Convolution layers apply a ﬁlter bank on their input.
The ﬁlter weights are shared over all spatial dimensions, meaning that for every location in the feature map the same set of ﬁlters are learned.
The distance inconsistentinreﬂectivity re-usability of weights causes a signiﬁcant reduction in the number of parameters compared to fully connected layers.
This allows deep convolutional neural networks to be trained successfully, in turn leading to a substantial performance boost in many computer vision applications.
The underlying premise of convolutional methods is that of translational symmetry, i.e. that features that have been learned in one region of the image are useful in other regions as well.
For applications such as face recognition which deal with aligned data, locally connected layers have proved to be advantageous [7, 8, 13].
These layers also apply a ﬁlter bank.
Contrary to convolutional layers, weights are not shared among the different locations in the feature map, allowing different sets of ﬁlters to be learned for every location in the input.
The spatial stationary assumption of convolutions does not hold for aligned images due to different local statistics in distant regions of the image.
In a projected LiDAR scan, the argument holds true for sensors that are mounted horizontally.
Each horizontal layer is ﬁxed at a certain vertical angle.
As the environment of the sensor is not invariant against rotations around this axis, this leads to different distance statistics in each vertical layer.
There exists some other works that exploit variability in depth along the vertical axis [2, 3, 5, 6, 11].
Though their work is conceptually similar to ours, they report improvements in contrast to this work.
However, these works do not include an ablation study on this particular architectural modiﬁcation independent of the other modiﬁcations proposed in their work.
This makes the evaluation not transparent about the actual inﬂuence of the adapted convolutions.
Therefore, this work presents the results of an independent ablation and shows that explicit modelling of varying statistical properties in CNNs is not necessary for LiDAR-base semantic segmentation.
Method
We consider an input feature map x of shape
[ H x , W x , C x ] , representing a cylindrical projection with height H x , width W x , and channels C x .
It is passed through a convolution layer which outputs another feature map y of shape [ H y , W y , C y ] .
A conventional 2D convolution layer computes the output y with a cross-correlation of the input x and a ﬁlter kernel k such that
[Equation]
where k has the shape [2 I + 1 , 2 J + 1 , C x , C y ] .
In a SLC layer, the ﬁlter kernel has the shape [2 I + 1 , 2 J + 1 , C x , C y , α ] where α is the number of components with { α ∈ N : 1 ≤ α ≤ H x } .
Each component is responsible for different parts along the vertical axis of the input (note that this concept can also be applied to the horizontal direction).
The output of the SLC is then given by
[Equation]
[Table] where α h = (cid:98) α · h/H x (cid:99) selects the respective ﬁlter-component depending on the vertical position h .
For α = H x , there is no weight sharing along the vertical axis, a new ﬁlter is used for every single data row.
For α = 1 , we obtain a regular convolution as deﬁned in Eq.
For values in between, the degree of weight sharing can be adapted to the desired application.
All equations omit padding, bias, and activation function for better readability.
Experiments
In this section, we investigate the introduction of SLC layers in various experiments.
We use the implementation of Milioto et al.
[9] 2 for our experiments.
All results are reported on the validation split of the SemanticKITTI dataset [1].
The input to the network is a two-channel image with the projected depth measurements and the respective reﬂecivity values.
Table 1 shows the overall results for two different base networks.
First, it shows that SLCs do not outperform normal convolutions.
Second, the performance decreases with increasing α .
Third, the above two points apply to both base networks, even though DarkNet21 has approximately 24.7M trainable parameters, whereas SqueezeSegV2 only has approximately 928.5k parameters.
When α increases with a factor of 2 , then also the number of trainable parameters increases by a factor of 2 (approximately).
Therefore, we conduct a second row of experiments, where we reduce the number of trainable parameters in the base network by decreasing the number of ﬁlters in each layer.
Naturally, we expect a network with lower capacity to perform worse.
Table 2 shows the semantic segmentation accuracy and mIoU performance for the DarkNet21 model.
The entries with the gray background mark those networks that have approximately the same number of trainable variables as the base network, since the modiﬁcation in α and output ﬁlter size cancel each other out.
Here again, an increase in α leads to decreased performance.
One has to note, that even if the number of parameters is the same for the gray cells, the increase in α only leads to more capacity over the spatial dimension of the feature maps, whereas larger output ﬁlters in general lead to more capacity over the depth of the network for the entire spatial extent.
Discussion
The experiments show that SLCs are not able to outperform normal convolutions and performance usually decreases with increasing α .
This effect is stronger for networks with a large number of parameters.
Therefore, we assume that normal convolution layers of adequate capacity can already handle the different statistical properties across the vertical spatial dimension.
This claim is supported by the ﬁndings of Kayhan et al.
[10] who show that convolutional layers exploit absolute spatial location.
Therefore CNNs are in fact not translation invariant which means a network with sufﬁcient capacity is able to learn even differing statistics over the vertical dimension of such a LiDAR scan. [Table]  Limitations and Future Work In contrast to other works
[Table]
we present an independent ablation on our proposed SLC layer.
However [Table] we still see the need for more extensive evaluations to obtain comprehensive evidence of the effects of SLCs. First, is must be clariﬁed whether the results are data dependent.
This can be achieved by using other datasets, such as nuScenes [4] and Waymo Open [12]
[Table]
or dense depth representations instead of range images.
Second, the reported results could be task-dependent and therefore SLC might yield different results for tasks, such as object detection or motion estimation.
Future work may include a soft version of the proposed SLC, where the α strips of data are not processed completely independently.
The output is then computed via a linear combination of multiple kernel operations, weighted with the vertical position of the respective ﬁlter.
Conclusion
This paper presented semi local convolutions (SLCs), a network layer that only has limited weight sharing along the vertical spatial dimension of the input.
Our experiments showed that using SLCs instead of normal convolutions decreases segmentation performance, especially when decreasing the amount of weight sharing.
Since normal convolution layers can already exploit spatial location information within the network, it is not necessary to explicitly address the large difference in appearance along the vertical axis of a LiDAR scan in a special layer.