Original: A benchmark with decomposed distribution shifts for 360 monocular depth estimation
Revised: A benchmark with decomposed distribution shifts for 360 monocular depth estimation

Original: Abstract
Revised: Abstract

Original: In this work we contribute a distribution shift benchmark for a computer visiontask; monocular depth estimation.
Revised: In this work we contribute a distribution shift benchmark for a computer vision task; monocular depth estimation.

Original: Our differentiation is the decomposition ofthe wider distribution shift of uncontrolled testing on in-the-wild data to threedistinct distribution shifts.
Revised: Our differentiation is the decomposition of the wider distribution shift of uncontrolled testing on in-the-wild data, to three distinct distribution shifts.

Original: Speciﬁcally, we generate data via synthesis and analyzethem to produce covariate (color input), prior (depth output) and concept (theirrelationship) distribution shifts.
Revised: Speciﬁcally, we generate data via synthesis and analyze them to produce covariate (color input), prior (depth output) and concept (their relationship) distribution shifts.

Original: We also synthesize combinations and show howeach one is indeed a different challenge to address, as stacking them producesincreased performance drops and cannot be addressed horizontally using standardapproaches.
Revised: We also synthesize combinations and show how each one is indeed a different challenge to address, as stacking them produces increased performance drops and cannot be addressed horizontally using standard approaches.

Original:  Introduction
Revised:  Introduction

Original: Data-driven methods are conditioned on the data which are available during the model developmenttime but are to be applied to in-the-wild real world data.
Revised: Data-driven methods are conditioned on the data which are available during the model development but are to be applied on real world data.

Original: Considering that the former data distributionis P s = σ ( P r ) , which is the result of applying a sampling function σ ( · ) to the real world distribution 
Revised: Considering that the former data distribution is P s = σ ( P r ) , which is the result of applying a sampling function σ ( · ) to the real world distribution P r .

Original: P r .
Revised: Typically,

Original: Typically, P s is separated into different splits P trn and P val , used to train the model and validateits behaviour respectively, with the latter process driving model selection.
Revised: P s is separated into different splits P trn and P val , used to train the model and validate its behaviour respectively, with the latter process driving model selection.

Original: A data distribution shiftcan be described as the condition where the joint distribution P of inputs and outputs differs betweenthe training and test stages [1].
Revised: A data distribution shift can be described as the condition where the joint distribution P of inputs and outputs differs between the training and test stages [1].

Original: This is an actual problem that many practical applications face, affecting their overall performance,robustness and reliability.
Revised: This is an actual problem that many practical applications face, affecting their overall performance, robustness and reliability.

Original: This phenomenon is more prominent in tasks where annotated datacollection is difﬁcult. This problem has been generally addressed in the literature as the domain shift[2] or the generalization[3] of data-driven models, or otherwise as out-of-distribution robustness [4].
Revised: The phenomenon is more prominent in tasks where annotated data collection is difﬁcult and has been generally addressed in the literature as the domain shift [2] or the generalization of data-driven models [3, 4] or otherwise as out-of-distribution robustness [5].

Original: More information about out-of-distribution (OOD) learning and generalization can be found in recentsurveys [5, 6].
Revised: More information about out-of-distribution (OOD) learning and generalization can be found in recent surveys [6, 7, 8].

Original: Up to now most works approach this problem in its general setting via zero-shot cross datasettransfer experiments that aim at assessing model performance under a general distribution shiftconsidering two different samplings P s 1 = σ 1 ( P r ) and P s 2 = σ 2 ( P r ) .
Revised: Up to now, most works approach this problem in its general setting via zero-shot cross-dataset transfer experiments that aim at assessing model performance under a general distribution shift, considering two different samplings P s 1 = σ 1 ( P r ) and P s 2 = σ 2 ( P r ) , as seen in Figure 1.

Original: A recent benchmark [7]provided simultaneously data for subpopulation shift, a special case of distribution shifts, as well as ageneric domain generalization shift across a number of datasets and tasks.
Revised: A recent benchmark [9] provided simultaneously data for sub-population shift, a special case of distribution shifts, and a generic domain generalization shift across a number of datasets and tasks.

Original: In this work, we contribute a novel benchmark for distribution shift performance assessment for acomputer vision task notorious for its complex data collection processes, monocular depth estimation.
Revised: In this work, we contribute a novel benchmark for distribution shift performance assessment, in the context of a computer vision task notorious for its complex data collection processes; monocular depth estimation.

Original: The novelty of our benchmark is the decomposition of distribution shifts into different testing splitsthat offer distribution shifts of various types either in isolated settings, or combined.
Revised: The novelty of our benchmark lies in the decomposition of generalized shift into components, expressed separately or in combination, via targeted test splits.

Original:  The Shifted Panoramas Dataset
Revised:  The Pano3D Dataset

Original: Our benchmark relies on two recent 3D building scan datasets, Matterport3D (M3D) [8] andGibsonV2(GV2)
Revised: Our benchmark relies on two recent 3D building scan datasets, Matterport3D (M3D) [10] and

Original: [9], as well as modern synthesisapproaches, to produce high quality spherical panoramascoupled withspherical depth maps.
Revised: GibsonV2 (GV2) [11], using modern synthesis to produce high quality spherical panoramas coupled with depth maps.

Revised: Sample images can be found in Figure 2.

Original: Speciﬁcally, we use M3D as a traditional in-distribution modeldevelopment dataset, and GV2 as a zero-shot cross dataset transfer and out-of-distribution benchmarkdataset.
Revised: Speciﬁcally, we use M3D as a traditional in-distribution model development dataset and GV2 as a zero-shot cross-dataset transfer, out-ofdistribution benchmark dataset.

Original: For M3D we consider its standard partitioning into train P M 3 Dtrn , validation P M 3 D val , and test P M 3 D tstsplits.
Revised: For M3D, we consider its standard partitioning into train P M 3 D , validation splits.

Original: The GV2 splits represent another sampling of the real world domain, or otherwise a zero-shotcross dataset transfer experiment.
Revised: The GV2 splits represent another sampling of the real world domain, or otherwise a zero-shot cross-dataset transfer experiment.

Original: Nonetheless, GV2 itself is partitioned into different splits, the tiny
Revised: Nonetheless, GV2 itself is partitioned into different splits, the tiny

Original: P GV 2 tiny , medium P GV 2 med , full P GV 2 full and fullplus P GV 2 full + splits 1 .
Revised: P GV 2 tiny , medium P GV 2 med , full P GV 2 full and fullplus P GV 2 fullplus splits 2 .

Original: After synthesizing coupled color anddepth panoramas for all splits of both datasets,and analyzing the different splits we  observe that it ispossible to decompose them into three core distribution shifts.
Revised: After synthesizing coupled color and depth panoramas for all splits of both datasets, we analyze them and observe that it is possible to decompose them into three core distribution shifts.

Revised: More on the characterization and decomposition of distribution shift can be seen on [12, 13, 14]:

Original: A covariate distribution shift represents a shift of the input domain, which in our case is the colorimage’s domain.
Revised: A covariate distribution shift represents a shift of the input domain, which in our case is the color image’s domain.

Original: As we rely on a synthesis  ( i.e. raytracing)approach to generate our data, we are alsoin control of the camera color transfer function.
Revised: As we rely on a synthesis approach ( i.e. raytracing) to generate our data, we are also in control of the camera color transfer function.

Original: Consequently, we  generate a shifted input distribution Pcov using the M3Dtest split P M3 D tst , where only the color domain has been shifted, illustrated in theﬁrst row of Figure 1.
Revised: Consequently, we can generate a shifted input distribution P using the M3D split P M tst , where only the color domain has been shifted.

Original: After examining the different splits’ statistics we also observe a prior probability distribution shiftmanifesting at the tiny, P GV 2 tiny , and medium, P GV 2 med , splits which corresponds to P prior , meaningthat the output depth distribution has shifted from the training one P M 3 D trn .The different depthdistributions between these splits are presented in Figure 1 in the second row.
Revised: After examining the different splits’ statistics we also observe a prior probability distribution shift manifesting at the tiny, P GV 2 tiny , and medium, P GV 2 med , splits which corresponds to P prior , meaning that the output depth distribution has shifted from the training one P M 3 D trn .

Original: Yet, the input (color)distribution is similar as the color camera transfer function is the same, and the context is alsopreserved to residential scenes.
Revised: Yet, the input (color) distribution is similar as the color camera transfer function is the same, and the context is also preserved to residential scenes.

Original: Finally, analysing  fullplus split, we observe a concept distribution shift , which is the shifted contextof the depicted scenes as evident in the third row of Figure 1.
Revised: Finally, analysing the fullplus split, we observe a concept distribution shift, which is the shifted context of the depicted scenes.

Original: While Matterport3D ( i.e. P M 3 D trn ) onlycontains indoor residential scenes, the fullplus split P GV 2 full + presents varying scenes like supermarkets,garages, under construction buildings, etc., corresponding to P conc .
Revised: While Matterport3D ( i.e. P M 3 D trn ) only contains indoor residential scenes, the fullplus split P GV 2 fullplus presents varying scenes like supermarkets, garages, under construction buildings, etc., corresponding to P conc .

Original: At the same time though, the input (color) and output (depth) distributions are preserved between P M 3 D trn and P conc .
Revised: At the same time though, the input (color) and output (depth) distributions are preserved between P M 3 D trn and P conc .

Original: Notably, ourbenchmark decomposes the wider domain shift into three distinct distribution shifts.
Revised: Notably, our benchmark decomposes the wider domain shift into three distinct distribution shifts.

Original: But since werely on synthesis processes, it is straightforward to combine distribution shifts, producing P covpriorand P covconc by re-rendering the corresponding splits with a shifted color transfer function, essentiallyadding a covariant shift to the prior and concept ones.
Revised: But since we rely on synthesis processes, it is straightforward to combine distribution shifts, producing P covprior and P covconc by re-rendering the corresponding splits with a shifted color transfer function, essentially adding a covariant shift to the prior and concept ones.

Original: This provides two extra combined distributionshift splits, with only the simultaneous prior and concept shifts missing.
Revised: This provides two extra combined distribution shift splits, with only the simultaneous prior and concept shifts missing.

Revised: Details can be seen in Figure 3.

Original: All  our data are publiclyavailable at http://anonymized.url.
Revised: All of our data are publicly available at: vcl3d.github.io/Pano3D.

Original:  Analysis
Revised:  Analysis

Original: We support our benchmark with a set of zero-cross dataset transfer experiments across the differentdistribution shifts.
Revised: We support our benchmark with a set of zero-shot cross-dataset transfer experiments across the different distribution shifts.

Original: We use a standard UNet [10] architecture training a supervised model with acomplex objective:
Revised: We use a standard UNet [15] architecture training a supervised model with a complex objective similar to [16]:

Original: [Equation]
Revised: [Equation]

Original: where L L 1 is an L1 loss, L cos is an angular loss deﬁned on the surface normals, L grad is the multiscale gradient matching loss from [11], and L vnl is the virtual normal loss [12].
Revised: where L L 1 is an L1 loss, L cos is an angular loss deﬁned on the surface normals, L grad is the multiscale gradient matching loss from [17], and L vnl is the virtual normal loss [18].

Original: All the independentterm weights are equally weighted, i.e. λ i = 1 . 

Revised: All the independent term weights are equally weighted, i.e. λ i = 1 .

Original: ∀
Revised: ∀

Original: i We initialize our model using [13] and optimizeit using a batch size of 4 and the Adam optimizer [14], using a learning rate of 0 . 0002 and its defaultmomentum parameter values.
Revised: i We initialize our model using [19] and optimize it using a batch size of 4 and the Adam optimizer [20], using a learning rate of 0 . 0002 and its default momentum parameter values.

Original: When training we only use P M 3 D trn and for all experiments we calculate standard metrics for depthestimation [15], as well as boundary [16, 17] and normals RMSE and accuracies [18].
Revised: When training we only use P M 3 D trn and for all experiments we calculate standard metrics for depth estimation [21], as well as boundary [22, 23] and normals RMSE and accuracies [24].

Original: We aggregateperformance across the different traits (direct depth, boundary and smoothness) them using a set ofindicators:
Revised: We aggregate performance across the different traits (direct depth, boundary and smoothness) them using a set of indicators:

Original: [Equation]
Revised: [Equation]

Original: where RMSE and RMSE o are the depth and normal angular errors respectively, dbe acc is theaccuracy boundary error from [16], F t is the F1-score for different edge thresholds from [17], andδ d and α a are the accuracy under thresholds for the depth and surface normals from [15] and [18]respectively.
Revised: where RMSE and RMSE o are the depth and normal angular errors respectively, dbe acc is the accuracy boundary error from [22], F t is the F1-score for different edge thresholds from [23], and δ d and α a are the accuracy under thresholds for the depth and surface normals from [21] and [24] respectively.

Original: Through these indicators we present an holistic view of how task performance is affected from thedifferent distribution shifts.
Revised: Through these indicators we present an holistic view of how task performance is affected from the different distribution shifts.

Original: In the following subsections, we examine isolated distribution shifts aswell as some of their combinations.
Revised: In the following subsections, we examine isolated distribution shifts as well as some of their combinations.

Original: Decomposed Shifts: Varying the input, output and combined domains.
Revised: Decomposed Shifts: Varying the input, output and combined domains.

Original: After training a supervised model on M3D’s train split, we examine its performance on the different distribution shifts wehave generated compared to that of the in-distribution test set.
Revised: After training a supervised model on M3D’s train split, we examine its performance on the different distribution shifts we have generated compared to that of the in-distribution test set.

Original: Figure 2 illustrates the results using theindicators from Eq.
Revised: Figure 4 illustrates the results using the indicators from Eq.

Original: We observe a performance drop for all distribution shifts, with the covariate(magenta box) and prior (orange box) being at about the same level, while the concept shift (cyanbox) presents the largest performance loss.
Revised: We observe a performance drop for all distribution shifts, with the covariate (magenta box) and prior (orange box) being at about the same level, while the concept shift (cyan box) presents the largest performance loss.

Original: At the same time, combining two distribution shifts hurtsperformance even more, as shown by the combined distribution shifts (violet box). 
Revised: At the same time, combining two distribution shifts hurts performance even more, as shown by the combined distribution shifts (violet box). [Figure] [Table] i b

Revised: Boundary Preservation Performance

Revised: [Figure]

Original: Photometric augmentations for covariate shift.
Revised: Photometric augmentations for covariate shift.

Original: Next, we examine the effect of  photometric augmentations ( i.e. brightness, contrast, hue shifts, and gamma corrections)  on the different distributionshifts, in isolation and in combination.
Revised: Next, we examine the effect of training with photometric augmentations ( i.e. brightness, contrast, hue shifts, and gamma corrections) and testing on the different (combined or not) distribution shifts.

Original: Figure 3 presents the results of a model trained with such augmentations when tested on the our benchmarks different splits.
Revised: Figure 5 presents the results comparing training with and without augmentations.

Original: Even though the fact that photometricaugmentations address camera domainshifts, or color transfer function shifts is already acknowledgedin the literature, through these experiments where the performance gains are only observed in thesplits where covariate shifts manifests, we verify the decomposition of our benchmark’s shifts.
Revised: It is generally acknowledged that photometric augmentations address camera domain or color transfer function shifts, and our experiments verify this, as performance gains are only observed in the splits where covariate shifts manifests.

Original: Pretraining for generalization boost.
Revised: Pretraining for generalization boost.

Original: It isalso usually acknowledged that pretraining on large imagedatasets like ImageNet [19] usually helps with the domain shift.
Revised: Another common assumption is that pretraining on large image datasets like ImageNet [25] helps address domain shifts.

Original: We perform another experiment,using the PNAS model [20], with all hyperparameters preserved, and train a model initialized  using 
Revised: We perform another experiment, this time using the PNAS model [26] with all hyperparameters preserved, and train one model initialized with pretrained weights and another one initialized using [19].

Revised: Figure 6 presents the results when tested on our benchmark’s different shifts.

Revised: Interestingly, we observe a performance boost in the splits where only a single distribution shift is present, where in contrast, the ones with two stacked distribution shifts show minimal gains.

Revised: This indicates that pretraining does not necessarily improve generalization – in the form of more transferable initial features – but, instead, only provides a better parameter initialization leading to higher quality parameters’ optimization.

Revised: The full array of the conducted experiments and their detailed results can be found in Table 1.

Original: [Figure]
Revised: [Figure]

Original: i d

Original: Direct Depth Performance
Revised: Direct Depth Performance

Original: [Figure]
Revised: [Figure]

Original: b

Original: Boundary Preservation Performance
Revised: Boundary Preservation Performance

Original: [Figure]
Revised: [Figure]

Original: i s

Original: Smoothness Performance
Revised: Smoothness Performance

Original: [Figure]

Original: pretrained weights, and one initialized using [13].

Original: Figure 4 presents the results of these experimentswhen tested on our benchmark’s different splits and distribution shifts. Interestingly [Figure] we observe aperformance boost in the splits where only a single distribution shift is present, where in contrast, theones with two simultaneous distribution shifts show minimal gains.
Revised: wo

Original: This indicates that pretraining

Original: [Figure]
Revised: [Figure]

Original: does not necessarily offer generalization – in the form of more transferable initial features – butinstead only helps as a good parameter initializer

Original: [Figure]
Revised: [Figure]

Original: compared to random initializations

Original: [Figure]

Original:  Conclusion
Revised:  Conclusion

Original: Distribution shifts are very important to the  application of data-driven methods.
Revised: Distribution shifting is pivotal to the real-world application of data-driven methods.

Original: In this work, wecontribute a distribution shift benchmark for an ill-posed dense computer vision task with difﬁcultdata collection processes.
Revised: In this work, we contribute a distribution shift benchmark for an ill-posed dense computer vision task, with notoriously difﬁcult data collection process.

Original: Seeking to facilitate future research towards addressing this challengingproblem, we decompose distribution shifts to input (covariate), output (prior) and their relationship(concept), and provide a baseline to build upon.
Revised: Seeking to facilitate future research towards addressing this challenging problem, we decompose distribution shift to input (covariate), output (prior) and their relationship (concept) [Figure] providing an experimental baseline for further experimentation and understanding [Figure]

Original: The baselines are accompanied byan analysis forthe simpler covariate shift case, using photometric augmentations, as well as the general case usingpretrained models.
Revised: [Figure] [Table] Acknowledgments and Disclosure of Funding This work was supported by the EC funded H2020 project ATLANTIS [GA 951900].