Self-supervised Monocular Trained Depth Estimation using Self-attention and Discrete Disparity Volume - ML Reproducibility Challenge
Anonymous author(s)
AfﬁliationAddress email
Reproducibility Summary
Depth estimation is one of the widely studied applications in the ﬁeld of computer vision as it captures the perception of the 3D world.
This knowledge can be used in applications like autonomous vehicles and obstacle warning systems
The
paper [1] proposes to estimate the depth by a self-supervised technique where the model is trained using a sequence of monocular input images
Scope of Reproducibility
The state-of-the-art results in depth estimation are mostly based on fully-supervised techniques.
Although recent worksin unsupervised techniques have shown promising results, the performance gap is still prominent and they mostly relyon stronger supervision signals (stereo-supervision or monocular+stereo supervision).
The paper proposes to close thisperformance gap with the fully-supervised methods using only the monocular sequence for training with the help ofadditional layers - self-attention and discrete disparity volume.
Methodology
The architecture proposed in the paper was implemented from scratch since the code is not available online.
Ablationstudy, additional experiments using hyperparameters were performed and the results obtained from reproduction werecompared with that claimed in the paper.
The experiments were trained and tested using GeForce GTX 1080 GPU.
Results
We reproduced the performance proposed in the paper to within 2% of the reported values.
Therefore, it can beinferred that the use of self-attention and discrete disparity volume helped in improving the performance using only themonocular sequence of images.
What was easy
Reproduction of the encoder in the proposed algorithm was easily handled.
The description of the self-attention layer iselaborate and could be conveniently reproduced.
What was difﬁcult
The unavailability of the code online made it inevitable for the reproduction of the algorithm from scratch.
The processof training using the ResNet101 encoder was time-consuming, however, the memory utilization was reduced to half dueto the usage of activated batch normalization.
Communication with original authors
The authors helped in understanding concepts like the usage of dilated convolution and discrete disparity volume. This in
Introduction
Monocular depth estimation is a technique of inferring the depth of objects from a single sequence of input images.
It isoften referred to as an ill-posed problem as more than one point in 3D converges to the same point in 2D. Traditional fullysupervised learning methods [2] [3] that have achieved state-of-the-art results in the ﬁeld of depth estimation, use groundtruth data obtained from LIDARs to estimate the depth map.
Acquiring ground truth data is a tedious task since theyrequire accurate instrumentation and synchronization of data streams from multiple sensors.
Therefore, unsupervisedtechniques are preferred.
In the unsupervised technique, we have two kinds - stereo learning and monocular learning.
Monocular learning uses simple readily available monocular cameras hence are preferred over stereo learning whichuses an expensive stereo camera.
In monocular technique, the depth information is extracted from the sequence ofimages separated across time.
Recent works [4] [5] on monocular depth estimation use dual networks - disparity andpose - to achieve promising results.
However, they mostly rely on stronger supervision signals(stereo-supervision ormonocular+stereo supervision).
This paper “Self-supervised Monocular Trained Depth Estimation using Self-attentionand Discrete Disparity Volume” [1], improves on the existing work by using only monocular sequence to train andclose the performance gap between self-supervised and fully-supervised models.
1.1 Background
Depth estimation is crucial to understand the 3D world for navigation assistance in computer vision and roboticssystems.
Several methods to estimate depth have been adopted over the years.
Initially, active sensors like LiDAR and
RADAR were used on the navigating agent [6].
However, due to its limited range, increased noise, power consumption,and cost, a deep convolutional neural network with passive sensors became popular.
Deep convolutional neural networkmethods can be mainly categorized into 3 major types - 1.
Fully supervised learning, 2.
Self-supervised stereo learning,and 3.
Self-supervised monocular learning.
Fully supervised methods require input images along with their corresponding depth maps for the network to train.
Obtaining a huge amount of ground truth depth data in all the environments where the navigating agent is deployedcan be tedious.
However, the performance of this method holds the current state-of-the-art values for depth estimation.
Next, we have self-supervised stereo learning which does not require ground truth data but instead relies on the leftand right view of the camera.
While self-supervised monocular learning is considered a good alternative because of itsease in collecting the input data, it poses many problems.
Depth estimation using monocular sequence is inherentlyambiguous and an ill-posed problem.
Recent advances leverage monocular images to estimate depth by adopting a dualnetwork.
One such architecturally advanced solution is proposed by this paper [1] to close the performance gap withfully supervised methods.
1.2 Summary of the Paper
This research [1] aims to outstrip the existing state-of-the-art depth estimation algorithm in terms of performance.
Toachieve this, two techniques are introduced.
The Self-attention layer is used to extract the global context of an imagewhich contrasts the traditionally used convolution layer that captures only the local context.
And the other techniqueused is the multi-scale discrete disparity volume decoder.
This technique is mainly used to obtain more robust andsharper depth maps from the network.
Discrete disparity volume helps in estimating pixel-wise uncertainty, which iscrucial for reﬁning depth estimation.
The paper adopts the pose estimating neural network and loss functions - photo-metric re-projection loss and edge-awaresmoothness loss - from Monodepth2 [5].
The depth network is used to predict the depth map D t of image I t .
Whilethe pose network is used to predict the transformation matrix having 6 degrees of freedom using image frames I tand
I t ‘ ∈ { I t − 1 , I t +1 } where I t ‘ are the temporally adjacent frames to I t .
Images I t ‘ are transformed to I t usingthe transformation matrix resulting in I t ‘ → t and the photo-metric re-projection loss is given by the L1 distance inpixel space between I t
and I t ‘ → t .
Edge-aware smoothness loss helps to predict smooth disparity estimates in localneighborhoods and penalizes drastic depth changes in ﬂat regions.
The total loss is equivalent to the sum of photo-metricre-projection and weighted edge-aware regularization.
And the auto-masking technique is used to preserve the staticscene-moving camera assumption.
The rest of the paper is organized as follows.
Section 2 gives an overview of the claims made in [1].
Section 3 elaborateson the methodology, computational and experimental requirements used for reproducing the proposed algorithm.
Section 4 contains the reproduced results claimed in the paper along with additional experimental results.
In Section 5we have the analysis of the reproduction.
Scope of reproducibility
The paper [1] proposes to achieve quality performance using only the monocular sequence for training using twotechniques namely self-attention and discrete disparity volume.
The following are the major claims made in the paper -
• Addition of discrete disparity volume and self-attention layers show steady improvement in all evaluationmeasures.
• The performance obtained from the proposed architecture closes the gap between self-supervised and fullysupervised models.
Methodology
The goal of this study is to validate the claims made by the original paper and extend the analysis to other datasetsand methodologies.
First, we implemented the approach with the help of the description provided in the paper, as theoriginal code is unavailable online.
The implementation of the self-attention layer in [7] and discrete disparity layer in[3] were useful for the reproduction.
3.1 Model descriptions
The baseline depth architecture is adopted from [5].
The depth architecture has two main parts – encoder and decoder.
The image sequence is fed as input to the ResNet101 encoder with atrous spatial pyramid pooling (also known as adilated convolution in [1]), which learns the features of the input image to generate an output that is 1/8th the size of theinput image.
ResNet101 use pre-trained ImageNet weights.
The encoder is adjoined with a self-attention layer.
Thislayer explores the global context of the input image, unlike convolution layers which extract only the local context ofthe image.
A convolution layer is used to reduce the channels from the attention layer to 128, which is the number ofbins used by discrete disparity volume.
The decoder comprises a multi-scale discrete disparity volume layer to obtain arobust and sharper depth estimate.
Additionally to reduce the exhaustive memory usage of ResNet101, activated batchnormalization [8] which fuses batch normalization with activation function is used to bring the memory usage down byup to 50%.
The loss functions used and the pose network is the same as the Monodepth2 [5] implementation.
3.2 Datasets
KITTI Raw [9] and Cityscapes [10] datasets were used in the experiments.
The architecture is trained and evaluatedusing the KITTI dataset.
The train and validation set are split using the traditional Eigen et al.
[2] method, whichgives 39,810 monocular training sequences and 4,424 validation sequences.
Training monocular images are reduced to640 x 192 pixels and the following data augmentations are performed with 50% probability - horizontal ﬂip, randomcontrast( ± 0 . 2 ), saturation( ± 0 . 2 ), hue jitter( ± 0 . 1 ), and brightness( ± 0 . 2 ).
The depth for this dataset is evaluated upto a ﬁxed range of 80m. This value is the same as that used in Monodepth2 and the original authors of paper [1]. Corresponding results are recorded in this reproducibility.
The method was quantitatively compared with the held-out test set of Cityscapes datasets.
3.3 Hyperparameters
The pose and depth network is jointly optimized using Adam Optimizer [11] with β 1 = 0 .
9 , β 2 = 0 .
999 and learningrate of 1 e − 4 .
The training process is executed for 20 epochs and a single learning rate decay of 1 e − 5 is applied after 15epochs.
And the ﬁnal loss is computed by setting the smoothness term λ
= 1 e − 3 .
3.4 Experimental setup and code
Monodepth2 [5] implementation was used as reference to perform ablation experiments.
The same environmental setup-
python=3.6.6, torch=0.4.1 and torchvision=0.2.1 - as Monodepth2 was used.
Initially, the self-attention layer wasincluded in the encoder.
This was followed by replacing the decoder with multi-scale discrete disparity volume.
Finally,
ResNet101 with atrous spatial pyramid pooling was used as an encoder in -place on ResNet18.
In the ﬁnal model proposed, the encoder is adopted from the OCNet [7] architecture.
ResNet101 is used with minormodiﬁcations by replacing the convolutions in the last two blocks with dilated convolutions of rate 2 and 4 respectively. This is then fed to the self-attention layer. The atrous spatial pyramid pooling module is the result of the concatenationof the output of self-attention layer along with four other dilated convolution layers of rates [1, 12, 24, 36]. This architecture of the encoder is not easily deduced from the overall architecture(Figure 2 in [1]) proposed in the paper.
Referring to OCNet [7] helps in better understanding.
The decoder adopts the concept of multi-channel ordinalregression from DORN [3].
The proposed architecture makes use of uniform discretization.
For this reproduction, thecode snippet for the same was provided by the authors of [1].
Evaluation metrics like absolute relative, square relative, and RMSE log loss measures are used.
The lower these metricvalues are the better it is.
And accuracy is also calculated with three different delta values.
An increase in accuracysuggests better performance.
The code and references for each of the layers mentioned above can be found at https://github.com/sjsu-smart-lab/Selfsupervised-Monocular-Trained-Depth-Estimation-using-Self-attention-and-Discrete-Disparity-Volum
3.5 Computational requirements
The proposed algorithm was implemented using Pytorch [12] framework and the experiments were trained and testedusing NVIDIA
Tesla P100 GPU and CPU environment - Intel Xeon E5-2660 v4 (2.0GHz, 35M Cache).
The ResNet18models were executed with a batch size of 8 while the ResNet101 models were executed with a batch size of 4.
Table 1shows the computational requirements like memory usage, training time and inference time on the KITTI Raw dataset[9].
We observe that as each additional layer is added the resource utilization increases substantially.
[Table]
The GPU inference time and training time with the addition of only the self-attention layer is less when comparedwith the addition of only discrete disparity volume.
However, the CPU inference time and the memory utilized withthe addition of only self-attention is higher when compared with the addition of only discrete disparity volume to thearchitecture.
The reduced memory utilization in the architecture with both self-attention and discrete disparity volumelayers is credited to the use of activated batch normalization.
State-of-the-art Results
The results were obtained after implementing with the help of the description given in the paper.
The evaluation metricsof the ﬁnal reproduced model are close to the values reported in the paper.
A subset of the attention maps obtained fromthe proposed methods is shown in Figure 1.
4.1 Results reproducing original paper
The following are the results obtained to validate the claims(Section 2) made in the paper.
All the experiments ontraining were performed on the KITTI Raw dataset [9].
4.1.1 Ablation Study
The performance after introducing each new layer - self-attention, discrete disparity volume and ResNet101 with atrousspatial pyramid pooling was recorded as part of the ablation study.
[Table]
ResNet101 w/ ASPP
[Table] Paper [1] [Table] From the overall reproduced model in Table 2, it can be inferred that the absolute relative loss is around 2% behind thevalue stated in the paper while the square relative loss value is improved by around 3%.
The other metric values areclose to the stated values.
Therefore, it can be concluded that the overall performance of the reproduction is almost inequivalence with the results proposed in the paper [1].
4.1.2 Quantitative Analysis of the Performance
The current state-of-the-art result in the ﬁeld of depth estimation is held by a fully-supervised method [3].
However,self-supervised monocular learning being more convenient has shown considerable performance gain.
Monodepth
[5],which previously held the highest state-of-the-art values for self-supervised monocular learning was about 60% behind
DORN [3].
The performance of the reproduced model surpasses Monodepth
[5], and is about 47% behind [3].
Thus, itcan be inferred that the use of ResNet101 with atrous spatial pyramid pooling, self-attention, and discrete disparity volume helped in closing the performance gap with fully-supervised methods using only the monocular sequence ofimages for training.
Method
[Table]
[-Monodepth
-]
4.1.3 Identifying sensitive regions
The paper proposes to achieve sharper results on sensitive regions like thinner structures such as poles and reﬂectivesurfaces.
From the Figure 3 this can validated.
The following additional experiments were conducted to further validate the results proposed in the paper.
4.2.1 Inference on Cityscapes dataset
The model obtained from training the proposed algorithm on KITTI Raw dataset [9] is tested on the held out test set ofthe Cityscapes [10] data to analyze the generalizability of the trained model.
The result of this experiment is recordedbelow (Table 4).
[Table]
Monodepth
[Table] Paper [1]
[Table]
From Table 4, it can be observed that like with the KITTI Raw dataset, the model reproduced from [1] perform betterthan Monodepth
[5] when tested using the Cityscapes dataset as well.
4.2.2 Comparison between Discretization Methods
The decoder network is adopted from the research proposed in DORN [3] where ordinal regression is used to predictdepth.
DORN proposes two types of discretization methods that can be used as part of the decoder i.e. uniformdiscretization (UD) and space increasing discretization (SID).
The following experiment is to compare the performancebetween these two discretization methods. [Table]
Discretization
[Table]
From Tables 5 and 6 it is observed that as mentioned in [3] there is no signiﬁcant difference in the performance betweenboth the discretization methods.
However, the training time, resource utilized and inference time of SID is slightly morewhen compared with that utilized by UD.
Most importantly UD is a single-line implementation using torch.linspacefunction while SID is not. Therefore the authors of [1] prefer UD over SID.
Discussion
Self-supervised monocular learning is widely appreciated for its ease in collecting input training data.
However, itsperformance falls back when compared with stereo learning and fully-supervised learning.
We would acknowledgethis paper for its novel contribution to close this performance gap through techniques like self-attention and discretedisparity volume.
5.1 Discrete Disparity Volume
The decoder of the architecture which constitutes multi-scale discrete disparity volume could be elaborated for betterunderstanding.
Discrete disparity volume is a strategy through which the continuous depth value is divided into discretevalues [3].
This process is referred to in [3] as ordinal regression.
Unlike a regular regression process where thedepth value at each pixel is predicted from the continuous range of values, ordinal regression is more like a multiclassclassiﬁcation process where the depth values are predicted from the collection of discrete values.
The discretizationprocess can be done using either space-increasing discretization(SID) or uniform discretization(UD).
In UD the rangeof continuous depth values [ α, β ] are equally divided into bins with discrete threshold values given by equation
[Equation]
However, since the error maximizes at higher depth values, UD tends to strengthen the loss at these large depth values.
Therefore, SID is introduced to down weight the larger depth values.
Discrete threshold values of SID is given by theequation
[Equation]
where t i ∈ { t 0 , t 1 , ..., t K } are discretization thresholds and K is the number bins needed (K=128, as given in paper).
Therefore, the research in [3] suggests that SID outperforms UD when used as a decoder in depth network.
However,the paper [1] proposes a decoder that makes use of UD to generate its performance.
Unlike in DORN [3], from theexperiments we conducted during the process of reproduction, it is observed that there is no signiﬁcant difference inperformance between UD and SID.
We believe that this difference between results of DORN [3] and the paper [1]merits for more research.
5.2 Observations
The paper proposes an advanced architectural enhancement to close the performance gap with fully-supervised methods.
Depth estimation is crucial to understand the surrounding environment.
Therefore, it is one of the important pieces ofinformation needed for autonomous navigation systems.
The improved precision obtained in this paper is beneﬁcial forautonomous navigation applications to be commercially viable.
However, this complex architecture proposed occupiesextensive computational resources and it also has an extended response time as shown in Table 1.
The performanceimprovement when qualitatively analyzed is better than the previous state-of-the-art by a few decimals.
When anapplication in hand is more towards assisting humans rather than making decisions, then a method with a simplerarchitecture can be preferred over the architecture proposed in the paper.
The previous state-of-the-result was held by Monodepth2 [5] This architecture comprises of ResNet18 encoder, depthdecoder, and the pose network.
It occupies 81% smaller memory when compared with the reproduced architecture.
Butthe performance of the reproduced model is higher than Monodepth2.
Therefore, appropriate models that fulﬁll therequirements of the application can be used.
For future work, we would like to repeat the experimentation with increased batch size and vary hyper-parameters likeweight decay to study the variations in performance.
Also, we would explore techniques to reduce the computationalfootprint and inference time of this complex architecture.
5.3 What was easy
The OCNet architecture [7] referenced by the authors could be adopted conveniently to implement the encoderarchitecture.
It enfolds ResNet101, atrous spatial pyramid pooling, and self-attention layers.
Ablation experiments with
ResNet18 could be performed faster as it is smaller when compared to ResNet101.
5.4 What was difﬁcult
The unavailability of the code online made it inevitable for the implementation of the algorithm from scratch.
Theprocess of training using the ResNet101 encoder was time-consuming, however, the memory utilization was reduced tohalf due to the usage of activated batch normalization.
The following are the concepts that were not straightforward inthe paper and required the author’s input to understand and implement -
• The implementation of the encoder with atrous spatial pyramid pooling cannot be perceived using Figure2 of original paper [1].
Reference to OCNet architecture as prompted by the authors helped in encoderimplementation.•
The description of discrete disparity volume was limited.
Reference to DORN [3] and inputs from authorsabout the UD usage helped implement the decoder.
5.5 Communication with original authors
Warm regards to the authors, Adrian Johnston and Gustavo Carneiro for guiding us through the process.
They helpedclarify the experiment settings and guided us with the right resources to accurately reproduce the experiments under thesame settings.
They pointed us to the OCNet architecture [7] which reduced the workload.
Also, they answered ourquestions on discrete disparity volume and also provided the code snippet for the same.