S EMANTICALLY -G UIDED R EPRESENTATION L EARNING FOR S ELF -S UPERVISED M ONOCULAR D EPTH
A BSTRACT
Self-supervised learning is showing great promise for monocular depth estimation, using geometry as the only source of supervision.
Depth networks are indeed capable of learning representations that relate visual appearance to 3D properties by implicitly leveraging category-level patterns.
In this work we investigate how to leverage more directly this semantic structure to guide geometric representation learning, while remaining in the self-supervised regime.
Instead of using semantic labels and proxy losses in a multi-task approach, we propose a new architecture leveraging ﬁxed pretrained semantic segmentation networks to guide self-supervised representation learning via pixel-adaptive convolutions.
Furthermore, we propose a two-stage training process to overcome a common semantic bias on dynamic objects via resampling.
Our method improves upon the state of the art for self-supervised monocular depth prediction over all pixels, ﬁne-grained details, and per semantic categories.
I NTRODUCTION
Accurate depth estimation is a key problem in computer vision and robotics, as it is instrumental for perception, navigation, and planning.
Although perceiving depth typically requires dedicated sensors (e.g., stereo rigs, LiDAR), learning to predict depth from monocular imagery can provide useful cues for a wide array of tasks (Michels et al., 2005; Kendall et al., 2018; Manhardt et al., 2019; Lee et al., 2019).
Going beyond supervised learning from direct measurements (Eigen et al., 2014), self-supervised methods exploit geometry as supervision (Guo et al., 2018; Pillai et al., 2019;
Zou et al., 2018; Yang et al., 2017), therefore having the potential to leverage large scale datasets of raw videos to outperform supervised methods (Guizilini et al., 2019).
Although depth from a single image is an ill-posed inverse problem, monocular depth networks are able to make accurate predictions by learning representations connecting the appearance of scenes and objects with their geometry in Euclidean 3D space.
Due to perspective, there is indeed an equivariance relationship between the visual appearance of an object in 2D and its depth, when conditioned on the object’s category .
For instance, a car 25 meters away appears smaller (on the image plane) than a car only 5 meters away but bigger than a truck 50 meters away.
Current depth estimation methods either do not leverage this structure explicitly or rely on strong semantic supervision to jointly optimize geometric consistency and a semantic proxy task in a multi-task objective (Ochs et al., 2019; Chen et al., 2019), thus departing from the self-supervised paradigm.
In this paper, we explore how we can leverage semantic information to improve monocular depth prediction in a self-supervised way .
Our main contribution is a novel architecture that uses a ﬁxed pre-trained semantic segmentation network to guide geometric representation learning in a selfsupervised monocular depth network.
In contrast to standard convolutional layers, our architecture uses pixel-adaptive convolutions (Su et al., 2019) to learn semantic-dependent representations that can better capture the aforementioned equivariance property.
Leveraging semantics may nonetheless introduce category-speciﬁc biases.
Our second contribution is a two-stage training process where we automatically detect the presence of a common bias on dynamic objects (projections at inﬁnity) and resample the training set to de-bias it.
Our method improves upon the state of the art in selfsupervised monocular depth estimation on the standard KITTI benchmark (Geiger et al., 2013), both on average over pixels, over classes, and for dynamic categories in particular.
R ELATED W ORK
Since the seminal work of Eigen et al.
(2014), substantial progress has been done to improve the accuracy of supervised depth estimation from monocular images, including the use of Conditional
Random Fields (CRFs) (Li et al., 2015), joint optimization of surface normals (Qi et al., 2018), fusion of multiple depth maps (Lee et al., 2018), and ordinal classiﬁcation (Fu et al., 2018).
Consequently, as supervised techniques for depth estimation advanced rapidly, the availability of largescale depth labels became a bottleneck, especially for outdoor applications.
Garg et al.
(2016) and
Godard et al.
(2017) provided an alternative self-supervised strategy involving stereo cameras, where
Spatial Transformer Networks (Jaderberg et al., 2015) can be used to geometrically warp, in a differentiable way, the right image into a synthesized left image, using the predicted depth from the left image.
The photometric consistency loss between the resulting synthesized and original left images can then be minimized in an end-to-end manner using a Structural Similarity term (Wang et al., 2004) and additional depth regularization terms.
Following Godard et al.
(2017) and Ummenhofer et al.
(2017), Zhou et al.
(2017) generalized this to the purely monocular setting, where a depth and a pose networks are simultaneously learned from unlabeled monocular videos.
Rapid progress in terms of architectures and objective functions (Yin & Shi, 2018; Mahjourian et al., 2018;
Casser et al., 2019; Zou et al., 2018; Klodt & Vedaldi, 2018; Wang et al., 2018; Yang et al., 2018) have since then turned monocular depth estimation into one of the most successful applications of self-supervised learning, even outperforming supervised methods (Guizilini et al., 2019).
The introduction of semantic information to improve depth estimates has been explored in prior works, and can be broadly divided into two categories.
The ﬁrst one uses semantic (or instance) information to mask out or properly model dynamic portions of the image, which are not accounted for in the photometric loss calculation.
G ¨ uney & Geiger (2015) leveraged object knowledge in a Markov
Random Field (MRF) to resolve stereo ambiguities, while Bai et al.
(2016) used a conjunction of instance-level segmentation and epipolar constraints to reduce uncertainty in optical ﬂow estimation.
Casser et al.
(2019) used instance-level masks to estimate motion models for different objects in the environment, and account for their external motion in the resulting warped image.
The second category attempts to learn both tasks in a single framework, and uses consistency losses to ensure that both are optimized simultaneously and regularize each other, so the information contained in one task can be transferred to improve the other.
For instance, Ochs et al.
(2019) estimated depth with an ordinal classiﬁcation loss similar to the standard semantic classiﬁcation loss, and used empirical weighting to combine them into a single loss for optimization.
Similarly, Chen et al.
(2019) used a uniﬁed conditional decoder that can generate either semantic or depth estimates, and both outputs are used to generate a series of losses also combined using empirical weighting to generate the ﬁnal loss to be optimized.
Our approach focuses instead on representation learning, exploiting semantic features into the selfsupervised depth network by using a pretrained semantic segmentation network to guide the generation of depth features.
This is done using pixel-adaptive convolutions, recently proposed in Su et al.
and applied to tasks such as depth upsampling using RGB images for feature guidance.
We show that different depth networks can be readily modiﬁed to leverage this semantic feature guidance, ranging from widely used ResNets (He et al., 2016) to the current state-of-the-art PackNet (Guizilini et al., 2019), with a consistent gain in performance across these architectures.
S ELF -S UPERVISED S TRUCTURE -F ROM -M OTION
Our semantically-guided architecture is developed within a self-supervised monocular depth estimation setting, commonly known as structure-from-motion (SfM).
Learning in a self-supervised structure-from-motion setting requires two networks: a monocular depth model f D : I → D , that outputs a depth prediction ˆ D = f D ( I ( p )) for every pixel p in the target image I ; and a monocular ego-motion estimator f x : ( I t , I S )
→ x t → S , that predicts the 6 DoF transformations for all s ∈ S given by x t → s =
( R t 0 1 ) ∈ SE(3) between the target image I t and a set of temporal context source images I s ∈ I S .
In all reported experiments we use I t − 1 and I t +1 as source images.
3. T HE S ELF -S UPERVISED O BJECTIVE L OSS
We train the depth and pose networks simultaneously, using the same protocols and losses as described in Guizilini et al.
Our self-supervised objective loss consists of an appearance matching term L p that is imposed between the synthesized ˆ I t and original I t target images, and a depth regularization term L s that ensures edge-aware smoothing in the depth estimates ˆ D t .
The ﬁnal objective loss is averaged per pixel, pyramid scale and image batch, and is deﬁned as:
[Equation]
where λ 1 is a weighting coefﬁcient between the photometric L p and depth smoothness L s loss terms.
Following Godard et al.
(2017) and Zhou et al.
, the similarity between synthesized ˆ I t and original I t target images is estimated using a Structural Similarity (SSIM) term (Wang et al., 2004) combined with an L1 loss term, inducing the following overall photometric loss:
[Equation]
In order to regularize the depth in low gradient regions, we incorporate an edge-aware term similar to Godard et al.
This loss is weighted for each of the pyramid levels, decaying by a factor of 2 on each downsampling, starting with a weight of 1 for the 0 th pyramid level.
[Equation]
We also incorporate some of the insights introduced in Godard et al.
(2018), namely auto-masking, minimum reprojection error, and inverse depth map upsampling to further improve depth estimation performance in our self-supervised monocular setting.
3. D EPTH AND P OSE N ETWORKS
Our baseline depth and pose networks are based on the PackNet architecture introduced by Guizilini et al.
(2019), which proposes novel packing and unpacking blocks to respectively downsample and upsample feature maps during the encoding and decoding stages.
This network was selected due to its state-of-the-art performance in the task of self-supervised monocular depth estimation, so we can analyze if our proposed architecture is capable of further improving the current state-of-the-art.
However, there are no restrictions as to which models our proposed semantically-guided architecture can be applied to, and in Section 5.4 we study its application to different depth networks.
S EMANTICALLY -G UIDED G EOMETRIC R EPRESENTATION L EARNING
In this section, we describe our method to inject semantic information into a self-supervised depth network via its augmentation with semantic-aware convolutions.
Our proposed architecture is depicted in Figure 2 and is composed of two networks: a primary one, responsible for the generation of depth predictions ˆ D
= f D ( I ( p )) ; and a secondary one, capable of producing semantic predictions.
Only the ﬁrst network is optimized during self-supervised learning; the semantic network is initialized from pretrained weights and is not further optimized.
This is in contrast to the common practice of supervised (ImageNet) pretraining of depth encoders (Godard et al., 2018; Casser et al., 2019;
Zou et al., 2018): here instead of ﬁne-tuning from pre-trained weights, we preserve these secondary weights to guide the feature learning process of the primary depth network.
Our approach also differs from learning without forgetting (Li & Hoiem, 2017) by leveraging ﬁxed intermediate feature representations as a way to maintain consistent semantic guidance throughout training.
4. S EMANTICALLY -G UIDED D EPTH F EATURES
We leverage the information from the pretrained semantic network in the depth network through the use of pixel-adaptive convolutions (Su et al., 2019).
They were recently proposed to address some limitations inherent to the standard convolution operation, namely its translation invariance making it content-agnostic.
While this signiﬁcantly reduces the number of parameters of the resulting network, this might also lead to sub-optimal solutions under certain conditions important for geometric representation learning.
For example, spatially-shared ﬁlters globally average the loss gradients over the entire image, forcing the network to learn weights that cannot leverage location-speciﬁc information beyond their limited receptive ﬁelds.
Content-agnostic ﬁlters are unable to distinguish between different pixels that are visually similar (i.e. dark areas due to shadows or black objects) or generalize to similar objects that are visually different (i.e. cars with varying colors).
In this work, we use pixel-adaptive convolutions to produce semantic-aware depth features , where the ﬁxed information encoded in the semantic network is used to disambiguate geometric representations for the generation of multi-level depth features.
As shown in Figure 2, we extract multi-level feature maps from the semantic network.
For each feature map, we apply a 3 × 3 and a 1 × 1 convolutional layer followed by Group Normalization (Wu & He, 2018) and ELU non-linearities (Clevert et al., 2016).
These processed semantic feature maps are then used as guidance on their respective pixel-adaptive convolutional layers, following the formulation proposed in Su et al.
[Equation]
In the above equation, f ∈ R D are processed features from the semantic network that will serve to guide the pixel-adaptive convolutions from the depth network, p = ( x, y ) T are pixel coordinates, with [ p i − p
j ] denoting 2D spatial offsets between pixels, W k × k are convolutional weights with kernel size k , Ω i deﬁnes a k × k convolutional window around i , v is the input signal to be convolved, and b ∈ R 1 is a bias term.
K is the kernel used to calculate the correlation between guiding features, here chosen to be the standard Gaussian kernel:
[Equation]
where Σ ij is the covariance matrix between features f i and f j , here chosen to be a diagonal matrix σ 2 · I D , with σ as an extra learnable parameter for each convolutional ﬁlter.
These kernel evaluations can be seen as a secondary set of weights applied to the standard convolutional weights, changing their impact on the resulting depth features depending on the content stored in the guiding semantic features.
For example, the information contained in depth features pertaining to the sky should not be used to generate depth features describing a pedestrian, and this behavior is now captured as a larger distance between their corresponding semantic features, which in turn produces smaller weights for that particular convolutional ﬁlter.
Note that the standard convolution can be considered a special case of the pixel-adaptive convolution, where ∀ ij, K ( f i , f j )
= 1 .
4. S EMANTIC G UIDANCE N ETWORK
As the secondary network used to provide semantic guidance for the generation of depth features, we use a Feature Pyramid Network (FPN) with ResNet backbone (Lin et al., 2017).
This architecture has been shown to be efﬁcient for both semantic and instance-level predictions towards panoptic segmentation (Kirillov et al., 2019;
Li et al., 2018; Xiong et al., 2019; Porzi et al., 2019).
While our proposed semantically-guided architecture is not restricted to any particular network, we chose this particular implementation to facilitate the future exploration of different sources for guidance information.
Architectural details follow the protocols described in Li et al.
(2018), and unless mentioned otherwise the same pretrained model was used in all reported experiments.
The semantic network is assumed ﬁxed, pretrained on a held out dataset different than the raw data used for selfsupervised learning, i.e. we do not require any semantic ground truth on the target dataset.
4. T WO -S TAGE T RAINING
One well-known limitation of the self-supervised photometric loss is its inability to model dynamic objects, due to a static world assumption that only accounts for camera ego-motion (Godard et al., 2018; Casser et al., 2019).
A resulting common failure mode is the inﬁnite depth problem, which is caused by the presence of objects moving at the same speed as the camera.
This typically causes distinct holes in the predicted depth maps, with arbitrarily large values where these objects should be.
This severely hinders the applicability of such models in real-world applications, particularly for automated driving, where the ability to detect and properly model dynamic objects is crucial.
Moreover, this limitation may be further accentuated in our proposed semantically-guided architecture, as the inﬁnite depth problem occurs mostly on dynamic categories (i.e. cars and motorcycles) and the semantic-aware features may reinforce this bias.
We propose a simple and efﬁcient two-stage training method to detect and remove this bias from the training set.
In the ﬁrst stage, we learn a standard depth network on all available training data.
This network, exhibiting the inﬁnite depth problem, is then used to resample the dataset by automatically ﬁltering out sequences with inﬁnite depth predictions that violate a basic geometric prior.
We indeed ﬁnd that depth predictions for pixels corresponding to the nearby ground plane are generally robust.
This enables getting a coarse estimate of the ground plane using RANSAC and detecting the number of pixels whose predicted depth projects them signiﬁcantly below the ground.
If that number is above a threshold, then the corresponding image is subsequently ignored (we found a conservative threshold of 10 to work well in all our experiments, ﬁltering out roughly 5% of the KITTI training dataset).
During the second stage, we retrain the network on the subsampled dataset (from scratch to avoid the previous local optimum).
As this subsampled dataset is de-biased, the network learns better depth estimates on dynamic objects.
This process can be repeated, but we ﬁnd that two stages are enough to remove any traces of inﬁnite depth in our experiments, as shown in Figure 3.
E XPERIMENTAL R ESULTS 5. D ATASETS
We use the standard KITTI benchmark (Geiger et al., 2013) for self-supervised training and evaluation.
More speciﬁcally, we adopt the training, validation and test splits used in Eigen et al.
(2014) with the pre-processing from Zhou et al.
(2017) to remove static frames, which is more suitable for
[Table]
monocular self-supervised learning.
This results in 39810 images for training, 4424 for validation, and 697 for evaluation.
Following common practice, we pretrain our depth and pose networks on the CityScapes dataset (Cordts et al., 2016), consisting of 88250 unlabeled images.
Unless noted otherwise, input images are downsampled to 640 x 192 resolution and output inverse depth maps are upsampled to full resolution using bilinear interpolation.
Our ﬁxed semantic segmentation network is pretrained on Cityscapes, achieving a mIoU of 75% on the validation set.
5. I MPLEMENTATION D ETAILS
We implement our models with PyTorch (Paszke et al., 2017) and follow the same training protocols of Guizilini et al.
(2019) when optimizing our depth and pose networks.
The initial training stage is conducted on the CityScapes dataset for 50 epochs, with a batch size of 4 per GPU and initial depth and pose learning rates of 2 · 10 − 4 and 5 · 10 − 4 respectively, that are halved every 20 epochs.
Afterwards, the depth and pose networks are ﬁne-tuned on KITTI for 30 epochs, with the same parameters and halving the learning rates after every 12 epochs.
This ﬁne-tuning stage includes the proposed architecture, where information from the ﬁxed semantic network, pretrained separately, is used to directly guide the generation of depth features.
There is no direct supervision at any stage during depth training, all semantic information is derived from the ﬁxed secondary network.
When pretraining the semantic segmentation network, we use a ResNet-50 backbone with Imagenet (Deng et al., 2009) pretrained weights and optimize the network for 48 k iterations on the CityScapes dataset with a learning rate of 0 .
01 , momentum of 0 . 9 , weight decay of 10 − 4 , and a batch size of 1 per GPU.
Random scaling between (0 . 7 , 1 . 3) , random horizontal ﬂipping, and a crop size of 1000 × 2000 are used for data augmentation.
We decay the learning rate by a factor of 10 at iterations 36 k and 44 k .
Once training is complete, the semantic segmentation network is ﬁxed and becomes the only source of semantic information when ﬁne-tuning the depth and pose networks on KITTI.
5. D EPTH E STIMATION P ERFORMANCE
Our depth estimation results are summarized in Table 1, where we compare our proposed architecture with other published works.
From these results we can see that the introduction of semanticallyguided geometric representation learning further improves upon the current state of the art in selfsupervised monocular depth estimation from Guizilini et al.
(2019), which served as our baseline.
Our approach also outperforms other methods that leverage semantic information by a substantial margin, even those using ground-truth KITTI semantic segmentation and depth labels during train-
[Table]
ing (Ochs et al., 2019).
Furthermore, in Figure 5 we also present qualitative results showing the improvements in depth estimation generated by our proposed framework, compared to our baseline.
Note how our semantically-guided architecture produces sharper boundaries and better object delineation, especially in structures further away or not clearly distinguishable in the input image
[Table]
5. A BLATIVE A NALYSIS 5.4. D IFFERENT D EPTH N ETWORKS
To better evaluate our main contribution, we provide an ablative analysis showing how it generalizes to different depth networks.
To this end, we consider two variations of the widely used ResNet architecture as the encoder for our depth network: ResNet-18 and ResNet-50 (the same pretrained semantic network was used in all experiments).
Depth estimation results considering these variations are shown in Table 2, where we can see that our proposed semantically-guided architecture is able to consistently improve the performance of different depth networks, for all considered metrics.
5.4. C LASS -S PECIFIC D EPTH P ERFORMANCE
To further showcase the beneﬁts of our semantically-guided architecture, we also provide classspeciﬁc evaluation metrics, as shown in Figure 4.
As we do not have ground-truth semantic segmentation for these images, we use the prediction of the semantic network to bin pixels per predicted category, and evaluate only on those pixels.
From these results we can see that our proposed architecture consistently improves depth performance for pixels across all predicted classes, especially those containing ﬁne-grained structures and sharp boundaries, e.g. poles and trafﬁc signs.
We also measure the impact of our two-stage training process, which is expected to address the inﬁnite depth problem in dynamic objects.
Although we ﬁnd the pixel-average difference in performance to not be signiﬁcant (see Table 2), there is a signiﬁcant improvement in class-average depth estimation, from 0 .
121 to 0 .
This is because the number of pixels affected by the inﬁnite depth problem is vastly smaller than the total number of pixels.
However, when considering class-average depth evaluation, the improvement over classes such as cars ( 0 .
200 to 0 . 177 ) and motorcycles ( 0 .to 0 . 069 ) becomes statistically signiﬁcant.
This further exempliﬁes the importance of ﬁne-grained metrics in depth evaluation, so these underlying behaviors can be properly observed and accounted for in the development of new techniques.
C ONCLUSION
This paper introduces a novel architecture for self-supervised monocular depth estimation that leverages semantic information from a ﬁxed pretrained network to guide the generation of multi-level depth features via pixel-adaptive convolutions.
Our monodepth network learns semantic-aware geometric representations that can disambiguate photometric ambiguities in a self-supervised learning structure-from-motion context.
Furthermore, we introduce a two-stage training process that resamples training data to overcome a common bias on dynamic objects resulting in predicting them at inﬁnite depths.
Our experiments on challenging real-world data shows that our proposed architecture consistently improves the performance of different monodepth architectures, thus establishing a new state of the art in self-supervised monocular depth estimation.
Future directions of research include leveraging other sources of guidance (i.e. instance masks, optical ﬂow, surface normals), as well as avenues for self-supervised ﬁne-tuning of the semantic network.
A P RE - TRAINING THE S EMANTIC S EGMENTATION N ETWORK
The introduction of a semantic segmentation network to the depth estimation task increases the depth estimation performance, however it also increases model complexity (e.g. number of trainable parameters).
To investigate that the increased performance for the depth estimation task is indeed due to the semantic features encoded in the secondary network, we perform an in-depth analysis (summarized in Table 3) where we explore the impact of pre-training the semantic segmentation network before it is used to guide the generation of depth features.
From these results we can see that the presence of semantic information encoded in the secondary network indeed leads to an increase in performance, and that ﬁne-tuning this secondary network for the speﬁcic task of depth estimation actually decreases performance.
In the ﬁrst two rows an untrained semantic network is utilized, with only its encoder initialized from ImageNet (Deng et al., 2009) weights.
Two different scenarios are explored: in the ﬁrst one (D) only the depth network is ﬁne-tuned in a self-supervised fashion, while in D+S both networks are ﬁne-tuned together in the same way.
As expected, using untrained features as guidance leads to signiﬁcantly worse results, since there is no structure encoded in the secondary network and the primary network needs to learn to ﬁlter out all this spurious information.
When both networks are ﬁne-tuned simultaneously, results improve because now the added complexity from the secondary network can be leveraged for the task of depth estimation, however there is still no improvement over the baseline.
Next, the semantic network was pre-trained on only half of the CityScapes (Cordts et al., 2016) dataset (samples chosen randomly), leading to a worse semantic segmentation performance (validation mIoU of around 70% vs. 75% for the fully trained one).
This partial pre-training stage was enough to enable the transfer of useful information between networks, leading to improvements over the baseline.
Interestingly, ﬁne-tuning both networks for the task of depth estimation actually hurt performance this time, which we attribute to forgetting the information contained in the secondary network, as both networks are optimized for the depth task.
When the semantic network is pretrained with all of CityScapes (last two rows), these effects are magniﬁed, with ﬁne-tuning only the depth network leading to our best reported performance (Table 1) and ﬁne-tuning both networks again leading to results similar to the baseline.
[Table]
B U NCERTAINTY AND G ENERALIZATION TO D IFFERENT O BJECTS
In a self-supervised setting, increasing the number of unlabeled videos used for depth training is expected to lead to an increasing specialization away from the domain in which the semantic network was pre-trained.
This might result in harmful guidance if our method is not robust to this gap.
However, our approach does not use semantic predictions directly, but rather the decoded features of the semantic network themselves, which represent general appearance information that should be more robust to this domain gap.
To validate our hypothesis, we further explore the impact of erroneous semantic information in the performance of our proposed semantically-guided depth framework.
In Figure 6 we present qualitative results highlighting situations in which our pretrained semantic network failed to generate correct semantic predictions for certain objects in the scene, and yet our proposed framework was still able to properly recover depth values for that portion of the environment.
These exemplify possible scenarios for erroneous semantic prediction.
• Imprecise boundaries: in the ﬁrst row, we can see that the semantic segmentation network does not correctly detect the trafﬁc sign, yet the semantically-guided depth network predicts its shape and depth accurately.
• Wrong classiﬁcation: in the second row, the truck was mistakenly classiﬁed as partially “road” and “building”, however our semantically-guided depth network was still able to properly recover its overall shape with sharp delineation that was not available from its semantic contour.
A similar scenario happens in the same image, with “fence” being partially labeled as “bicycle”.
• Missing ontology: there is no “trash can” class on the CityScapes ontology, however in the third row our semantically-guided depth network was able to correctly reconstruct such object even though it was classiﬁed as “fence”, similarly to its surroundings.
• Object Hallucination: in the fourth row, the contour of a “person” was erroneously introduced in the image and correctly removed by our semantically-guided framework.
These examples are evidence that our proposed framework is able to reason over the uncertainty inherent to semantic classiﬁcation, leveraging this information when accurate to achieve the results reported in this paper, but also discarding it if necessary to generate a better reconstruction according to the self-supervised photometric loss.
C G ENERALIZATION TO D IFFERENT D ATASETS
In the previous sections, we show that our proposed framework is robust to a degraded semantic network, both by pretraining the semantic network with fewer annotated labels (Appendix A) and also by providing evidence that the depth network is able to reason over erroneous predictions to still generate accurate reconstructions (Appendix B).
We now go one step further and analyze how our proposed semantically-guided framework generalizes to a dataset that was used neither during pre-training nor for ﬁne-tuning.
To this end, we evaluate our KITTI depth model on the recently released NuScenes dataset (Caesar et al., 2019).
The ofﬁcial NuScenes validation split is used, containing 6019 images from the front camera with ground-truth depth maps generated by LiDAR reprojection.
Results presented in Table 4 provide additional evidence that our method indeed results in generalization improvements, even on signiﬁcantly different data from different platforms and environments (Karlsruhe, Germany for KITTI vs Boston, USA and Singapore for NuScenes), outperforming the state of the art methods and our baseline (Guizilini et al., 2019).
[Table]