Original: VA-D EPTH N ET : A V ARIATIONAL A PPROACH TO S INGLE
Revised: VA-D EPTH N ET : A V ARIATIONAL A PPROACH TO S INGLE

Original: I MAGE D EPTH P REDICTION
Revised: I MAGE D EPTH P REDICTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We introduce VA-DepthNet, a simple, effective, and accurate deep neural network approach for a single image depth prediction (SIDP) problem.
Revised: We introduce VA-DepthNet, a simple, effective, and accurate deep neural network approach for a single image depth prediction (SIDP) problem.

Original: The proposed approach advocate using classical first-order variational constraints for this problem.
Revised: The proposed approach advocate using classical first-order variational constraints for this problem.

Original: While state-of-the-art deep neural network methods for SIDP learn the scene depth from images in a supervised setting they often overlook the invaluable invariances and priors in the rigid scene space, such as the regularity of the scene.
Revised: While state-of-the-art deep neural network methods for SIDP learn the scene depth from images in a supervised setting, they often overlook the invaluable invariances and priors in the rigid scene space, such as the regularity of the scene.

Original: The paper’s main contribution is to reveal the benefit of classical and well-founded variational constraints in the neural network design for the SIDP task.
Revised: The paper’s main contribution is to reveal the benefit of classical and well-founded variational constraints in the neural network design for the SIDP task.

Original: It is shown that imposing first-order variational constraints in the scene space together with popular encoder-decoder-based network architecture design provides excellent results for the supervised SIDP task.
Revised: It is shown that imposing first-order variational constraints in the scene space together with popular encoder-decoder-based network architecture design provides excellent results for the supervised SIDP task.

Original: The imposed first-order variational constraint makes the network aware of the depth gradient in the scene space, i e., regularity 1 .
Revised: The imposed first-order variational constraint makes the network aware of the depth gradient in the scene space, i.e., regularity 1 .

Original: The paper demonstrates the usefulness of the proposed approach via extensive evaluation and ablation analysis over several benchmark datasets, such as KITTI, NYU
Revised: The paper demonstrates the usefulness of the proposed approach via extensive evaluation and ablation analysis over several benchmark datasets, such as KITTI, NYU

Original: Depth V2, and SUN RGB-D. The VA-DepthNet at test time shows considerable improvements in depth prediction accuracy compared to the prior art and is accurate also at high-frequency regions in the scene space.
Revised: Depth V2, and SUN RGB-D. The VA-DepthNet at test time shows considerable improvements in depth prediction accuracy compared to the prior art and is accurate also at high-frequency regions in the scene space.

Original: At the time of submission, our method—labeled as VA-DepthNet, when tested on the KITTI official depthprediction evaluation set, indexed second on the leader board and our accuracy is top performing among the published method 2 3 .
Revised: At the time of submission, our method—labeled as VA-DepthNet, when tested on the KITTI official depthprediction evaluation set, indexed second on the leader board, and our accuracy is top performing among the published method 2 3 .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Over the last decade, neural networks have introduced a new prospect for the 3D computer vision field.
Revised: Over the last decade, neural networks have introduced a new prospect for the 3D computer vision field.

Original: It has led to significant progress on many long-standing problems in this field, such as multiview stereo (Wang et al., 2021), vision-based simultaneous localization and mapping (Teed & Deng, 2021), novel view synthesis (Mildenhall et al., 2021) and others.
Revised: It has led to significant progress on many long-standing problems in this field, such as multiview stereo (Wang et al., 2021), vision-based simultaneous localization and mapping (Teed & Deng, 2021), novel view synthesis (Mildenhall et al., 2021) and others.

Original: Among several 3D vision problems, one of the challenging, if not impossible, to solve is the single image depth prediction (SIDP) problem.
Revised: Among several 3D vision problems, one of the challenging, if not impossible, to solve is the single image depth prediction (SIDP) problem.

Original: SIDP is indeed ill-posed—in a strict geometric sense, presenting an extraordinary challenge to reliably solving this inverse problem.
Revised: SIDP is indeed ill-posed—in a strict geometric sense, presenting an extraordinary challenge to reliably solving this inverse problem.

Original: Moreover, since we do not have access to multi-view images, it is hard to constrain this problem via well-proven geometric constraints (Hartley & Zisserman, 2003).
Revised: Moreover, since we do not have access to multi-view images, it is hard to constrain this problem via well-proven geometric constraints (Hartley & Zisserman, 2003).

Original: Accordingly, this problem generally narrow-down to an ambitious fitting problem, and therefore, learning techniques provide a convenient way to predict an acceptable solution to this problem (Yuan et al., 2022; Yin et al., 2019).
Revised: Accordingly, this problem generally narrow-down to an ambitious fitting problem, and therefore, learning techniques provide a convenient way to predict an acceptable solution to this problem (Yuan et al., 2022; Yin et al., 2019).

Original: Impressive earlier methods use Markov Random Field (MRF) to model monocular cues and the relation between several over-segmented image parts (Saxena et al., 2007; 2008).
Revised: Impressive earlier methods use Markov Random Field (MRF) to model monocular cues and the relation between several over-segmented image parts (Saxena et al., 2007; 2008).

Original: Nevertheless, with the recent surge in neural network architectures (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; He et al., 2016), which has an extraordinary capability to perform complex regression, many current works use deep learning to solve SIDP and have demonstrated high-quality results (Yuan et al., 2022;
Revised: Nevertheless, with the recent surge in neural network architectures (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; He et al., 2016), which has an extraordinary capability to perform complex regression, many current works use deep learning to solve SIDP and have demonstrated high-quality results (Yuan et al., 2022;

Original: Aich et al., 2021;
Revised: Aich et al., 2021;

Original: Bhat et al., 2021; Eigen et al., 2014; Fu et al., 2018; Lee et al., 2019; 2021).
Revised: Bhat et al., 2021; Eigen et al., 2014; Fu et al., 2018; Lee et al., 2019; 2021).

Original: Popular recent methods for SIPD are, of course, supervised yet used less in real-world[Figure] applications as compared to popular geometric multiple view methods (Labbé & Michaud, 2019;
Revised: Popular recent methods for SIPD are, of course, supervised yet used less in real-world applications as compared to popular geometric multiple view methods (Labbé & Michaud, 2019;

Original: Müller et al., 2022).
Revised: Müller et al., 2022).

Original: Nonetheless, a good solution to SIDP is highly desirable in robotics (Tateno et al., 2017), virtual-reality (Hoiem et al., 2005), augmented reality (Du et al., 2020), view synthesis (Hoiem et al., 2005) and other related vision tasks (Liu et al., 2019).
Revised: Nonetheless, a good solution to SIDP is highly desirable in robotics (Tateno et al., 2017), virtual-reality (Hoiem et al., 2005), augmented reality (Du et al., 2020), view synthesis (Hoiem et al., 2005) and other related vision tasks (Liu et al., 2019).

Original: In this paper, we advocate that despite the supervised approach being encouraging for the SIDP task, the restriction to study further and improve the nature of the solution must not wholly rely on the increase in the size and quality of the datasets.
Revised: In this paper, we advocate that despite the supervised approach being encouraging for the SIDP task, the restriction to study further and improve the nature of the solution must not wholly rely on the increase in the size and quality of the datasets.

Original: Instead, finding helpful geometric cues and scene priors that could help improve the SIDP results must be investigated.
Revised: Instead, finding helpful geometric cues and scene priors that could help improve the SIDP results must be investigated.

Original: Not that scene priors have not been studied to improve SIDP accuracy in the past.
Revised: Not that scene priors have not been studied to improve SIDP accuracy in the past.

Original: For instance, Chen et al.
Revised: For instance, Chen et al.

Original: (2016) uses pairwise ordinal relations between points to learn scene depth.
Revised: (2016) uses pairwise ordinal relations between points to learn scene depth.

Original: On the contrary, Yin et al.
Revised: On the contrary, Yin et al.

Original: (2019) uses surface normal as an auxiliary loss to improve the performance.
Revised: (2019) uses surface normal as an auxiliary loss to improve the performance.

Original: Other heuristic approaches, such as Qi et al.
Revised: Other heuristic approaches, such as Qi et al.

Original: (2018), jointly exploit the depth-to-normal relation for recovering scene depth and surface normal.
Revised: (2018), jointly exploit the depth-to-normal relation for recovering scene depth and surface normal.

Original: Yet, such state-of-the-art SIDP methods have limitations; for example, Chen et al.
Revised: Yet, such state-of-the-art SIDP methods have limitations; for example, Chen et al.

Original: (2016) ordinal relation for learning over-smooth the depth prediction results, thereby failing to preserve high-frequency surface details.
Revised: (2016) ordinal relation for learning over-smooth the depth prediction results, thereby failing to preserve high-frequency surface details.

Original: Conversely, Yin et al.
Revised: Conversely, Yin et al.

Original: (2019) relies on good depth map prediction from a deep network and the idea of virtual normal, which is computed by randomly sampling three non-colinear points with large distances hence complex and heuristic in nature.
Revised: (2019) relies on good depth map prediction from a deep network and the idea of virtual normal, which is computed by randomly sampling three non-colinear points with large distances hence complex and heuristic in nature.

Original: As of Qi et al.
Revised: As of Qi et al.

Original: (2018), it uses depth and normal consistency, which is good, yet it requires good depth map initialization.
Revised: (2018), it uses depth and normal consistency, which is good, yet it requires good depth map initialization.

Original: Further, depth is dependent on the structure information of the scene geometry.
Revised: Further, depth is dependent on the structure information of the scene geometry.

Original: In contrast, surface normal is local and can be computed using small neighborhood relations.
Revised: In contrast, surface normal is local and can be computed using small neighborhood relations.

Original: Thus, optimally exploiting joint normal and depth constraint effectiveness is challenging.
Revised: Thus, optimally exploiting joint normal and depth constraint effectiveness is challenging.

Original: This brings us to the point that further generalization of the regression-based SIDP pipeline is required.
Revised: This brings us to the point that further generalization of the regression-based SIDP pipeline is required.

Original: As mentioned before, existing approaches in this direction have limitations and are complex.
Revised: As mentioned before, existing approaches in this direction have limitations and are complex.

Original: In this paper, we propose a simple approach that provides better depth accuracy and generalize well acrossthe different scene (see Fig.1).
Revised: In this paper, we propose a simple approach that provides better depth accuracy and generalize well across different scene.

Original: To this end, we resort to the physics of variation (Mollenhoff et al., 2016; Oswald et al., 2012) in the neural network design for better generalization of the SIDP network, which by the way, keeps the essence of affine invariance (Yin et al., 2019). 
Revised: To this end, we resort to the physics of variation (Mollenhoff et al., 2016; Chambolle et al., 2010) in the neural network design for better generalization of the SIDP network, which by the way, keeps the essence of affine invariance (Yin et al., 2019). An image of a general scene—indoor or outdoor, has a lot of spatial regularity.

Revised: And therefore, introducing a variational constraint provides a convenient way to ensure spatial regularity and preserve information related to the scene discontinuities (Chambolle et al., 2010).

Original: Our proposed network is trained in a fully-supervised manner while encouraging the network to be mindful of the scene regularity where the variation in the depth gradient is large (refer Sec.3.1).
Revised: Consequently, the proposed network is trained in a fully-supervised manner while encouraging the network to be mindful of the scene regularity where the variation in the depth gradient is large (refer Sec.3.1).

Original: In simple terms, depth regression must be more than parameter fitting, and at some point, a mindful decision must be made—either by imaging features or by scene depth gradient variation, or both.
Revised: In simple terms, depth regression must be more than parameter fitting, and at some point, a mindful decision must be made—either by imaging features or by scene depth gradient variation, or both.

Original: As we demonstrate later in the paper, such an idea boosts the network’s depth prediction overall accuracy while preserving the high-frequency and low-frequency scene details (see Fig.2).
Revised: As we demonstrate later in the paper, such an idea boosts the network’s depth prediction overall accuracy while preserving the high-frequency and low-frequency scene details (see Fig.1).

Original: Our introduced neural network for SIDP disentangles the absolute scale from the metric depth and models unscaled depth map as the optimal solution to the pixel-level variational constraints via weighted first-order difference respecting the neighboring pixel depth gradient.
Revised: Our introduced neural network for SIDP disentangles the absolute scale from the metric depth and models unscaled depth map as the optimal solution to the pixel-level variational constraints via weighted first-order difference respecting the neighboring pixel depth gradient.

Original: Compared to previous methods, the network task has been shifted away from pixel-wise metric depth learning to learning the first-order difference of the scene, which alleviates the scale ambiguity and favors scene regularity.
Revised: Compared to previous methods, the network task has been shifted away from pixel-wise metric depth learning to learning the first-order difference of the scene, which alleviates the scale ambiguity and favors scene regularity.

Original: To realize that, we first employ neural network to predict the first-order difference of the depth map.
Revised: To realize that, we initially employ neural network to predict the first-order difference of the depth map.

Original: Then, we construct the partial differential equations representing the variational constraints by reorganizing the differences into a large matrix, i.e., an over-determined system of equations.
Revised: Then, we construct the partial differential equations representing the variational constraints by reorganizing the differences into a large matrix, i.e., an over-determined system of equations.

Original: Further, the network learns a weight matrix to eliminate redundant equations that does[Figure] not favor the introduced first-order difference constraint.
Revised: Further, the network learns a weight matrix to eliminate redundant equations that does not favor the introduced first-order difference constraint.

Original: Finally, the closed-form depth map solution is recovered via simple matrix operations.
Revised: Finally, the closed-form depth map solution is recovered via simple matrix operations.

Original: When tested on the KITTI official (Geiger et al., 2012) and NYU Depth V2 (Silberman et al., 2012) test sets, our method supersedes prior art depth prediction accuracy by a large margin.
Revised: When tested on the KITTI official (Geiger et al., 2012) and NYU Depth V2 (Silberman et al., 2012) test sets, our method supersedes prior art depth prediction accuracy by a large margin.

Original: Moreover, our pre-trained model on the NYU Depth V2 train set generalizes better on SUN RGB-D test set.
Revised: Moreover, our pre-trained model on the NYU Depth V2 train set generalizes better on SUN RGB-D test set.

Original:  P RIOR W ORK
Revised:  P RIOR W ORK

Original: Predicting the scene’s depth from a single image is a long-standing problem in computer vision, and impressive classical work exists to solve it.
Revised: Predicting the scene’s depth from a single image is a long-standing problem in computer vision, and impressive classical work exists to solve it.

Original: Nevertheless, in this work, we focus on a fullysupervised approach, and therefore, we discuss the related works that directly relate to the proposed approach.
Revised: Nevertheless, in this work, we focus on a fully- [Figure] supervised approach, and therefore, we discuss the related works that directly relate to the proposed approach.

Original: Broadly, we divide the popular supervised SIDP methods into three sub-categories.
Revised: Broadly, we divide the popular supervised SIDP methods into three sub-categories.

Original: (i) Depth Learning using Ranking or Ordinal Relation Constraint.
Revised: (i) Depth Learning using Ranking or Ordinal Relation Constraint.

Original: Zoran et al.
Revised: Zoran et al.

Original: and
Revised: and

Original: Chen et al.
Revised: Chen et al.

Original: (2016) argue that the ordinal relation between points is easier to learn than the metric depth.
Revised: (2016) argue that the ordinal relation between points is easier to learn than the metric depth.

Original: To this end, Zoran et al.
Revised: To this end, Zoran et al.

Original: (2015) proposes constrained quadratic optimization while Chen et al.
Revised: (2015) proposes constrained quadratic optimization while Chen et al.

Original: (2016) relies on the variation of the inception module to solve the problem.
Revised: (2016) relies on the variation of the inception module to solve the problem.

Original: Later, Xian et al.
Revised: Later, Xian et al.

Original: (2020) propose structure-guided sampling strategies for point pairs to improve training efficiency.
Revised: (2020) propose structure-guided sampling strategies for point pairs to improve training efficiency.

Original: Recently, Lienen et al.
Revised: Recently, Lienen et al.

Original: (2021) elaborate on the use of listwise ranking method based on the Plackett-Luce model (Luce, 2012).
Revised: (2021) elaborate on the use of listwise ranking method based on the Plackett-Luce model (Luce, 2012).

Original: The drawback of such approaches is that the ordinal relationship and ranking over smooth the depth solution making accurate metric depth recovery challenging.
Revised: The drawback of such approaches is that the ordinal relationship and ranking over smooth the depth solution making accurate metric depth recovery challenging.

Original: (ii) Depth Learning using Surface Normal Constraint.
Revised: (ii) Depth Learning using Surface Normal Constraint.

Original: Hu et al.
Revised: Hu et al.

Original: (2019) introduce normal loss in addition to the depth loss to overcome the distorted and blurry edges in the depth prediction.
Revised: (2019) introduce normal loss in addition to the depth loss to overcome the distorted and blurry edges in the depth prediction.

Original: Yin et al.
Revised: Yin et al.

Original: (2019) propose the concept of virtual normal to impose 3D scene constraint explicitly and capture the long-range relations in the depth prediction.
Revised: (2019) propose the concept of virtual normal to impose 3D scene constraint explicitly and capture the long-range relations in the depth prediction.

Original: The long-range dependency in 3D is introduced via random sampling of three non-colinear points at a large distance from the virtual plane.
Revised: The long-range dependency in 3D is introduced via random sampling of three non-colinear points at a large distance from the virtual plane.

Original: Lately, Long et al.
Revised: Lately, Long et al.

Original: (2021) proposed an adaptive strategy to compute the local patch surface normals at train time from a set of randomly sampled candidates and overlooks it during test time.
Revised: (2021) proposed an adaptive strategy to compute the local patch surface normals at train time from a set of randomly sampled candidates and overlooks it during test time.

Original: (iii) Depth Learning using other Heuristic Refinement Constraint.
Revised: (iii) Depth Learning using other Heuristic Refinement Constraint.

Original: There has been numerous works attempt to refine the depth prediction as a post-processing step.
Revised: There has been numerous works attempt to refine the depth prediction as a post-processing step.

Original: Liu et al. (2015), Li et al.
Revised: Liu et al. (2015), Li et al.

Original: (2015) and Yuan et al.
Revised: (2015) and Yuan et al.

Original: (2022) propose various approaches to utilize the Conditional Random Fields (CRF) to smooth the depth map.
Revised: (2022) propose various approaches to utilize the Conditional Random Fields (CRF) to smooth the depth map.

Original: Lee et al.
Revised: Lee et al.

Original: and Patil et al.
Revised: and Patil et al.

Original: (2022) utilize the planar assumptions to regularize the predicted depth map.
Revised: (2022) utilize the planar assumptions to regularize the predicted depth map.

Original: Qi et al.
Revised: Qi et al.

Original: (2018) adopt an auxiliary network to predict the surface normal, and then refine the predicted depth map following their proposed heuristic rules.
Revised: (2018) adopt an auxiliary network to predict the surface normal, and then refine the predicted depth map following their proposed heuristic rules.

Original: There are mainly two problems with such approaches: Firstly, these approaches rely on a good depth map initialization.
Revised: There are mainly two problems with such approaches: Firstly, these approaches rely on a good depth map initialization.

Original: Secondly, the heuristic rules and the assumptions might result in over-smoothed depth values at objects boundaries. 
Revised: Secondly, the heuristic rules and the assumptions might result in over-smoothed depth values at objects boundaries. Meanwhile, a few works, such as Ramamonjisoa et al. (2020), Cheng et al. , Li et al.

Original: In this paper, we proffer the benefits of variational approach to overcome the limitations with the existing state-of-the-art methods.
Revised: (2017) were proposed in the past with similar inspirations. Ramamonjisoa et al. (2020), Cheng et al. (2018) methods are generally motivated towards depth map refinement predicted from an off-the-shelf network.

Revised: On the other hand, Cheng et al.

Revised: (2018) proposes to use an affinity matrix that aims to learn the relation between each pixel’s depth value and its neighbors’ depth values.

Revised: However, the affinity matrix has no explicit supervision, which could lead to imprecise learning of neighboring relations providing inferior results.

Revised: On the contrary, our approach is mindful of imposing the first-order difference constraint leading to better performance.

Revised: Earlier, Li et al.

Revised: (2017) proposed two strategies for SIDP, i.e., fusion in an end-to-end network and fusion via optimization.

Revised: The end-to-end strategy fuses the gradient and the depth map via convolution layers without any constrain on convolution weights, which may not be an apt choice for a depth regression problem such as SIDP.

Revised: On the other hand, the fusion via optimization strategy is based on a non-differentiable strategy, leading to a nonend-to-end network loss function.

Original: The proposed method can provide excellent depth prediction without impractical requirements such as good depth initialization, planar scene assumptions and others.
Revised: Contrary to that, our method is well-constrained and performs quite well with a loss function that helps end-to-end learning of our proposed network. Not long ago, Lee & Kim (2019) proposed to estimate relative depths between pairs of images and ordinary depths at a different scale. By exploiting the rank-1 property of the pairwise comparison matrix, it recovers the relative depth map. Later, relative and ordinary depths are decomposed and fused

Original: [Figure]
Revised: [Figure]

Revised: to recover the depth.

Revised: On a slightly different note, Lee & Kim (2020) studies the effectiveness of various losses and how to combine them for better monocular depth prediction.

Revised: To sum up, our approach allows learning of confidence weight to select reliable gradient estimation in a fully differentiable manner.

Revised: Further, it proffers the benefits of the variational approach to overcome the limitations of the existing state-of-the-art methods.

Revised: More importantly, the proposed method can provide excellent depth prediction without making extra assumptions such as good depth initialization, piece-wise planar scene, and assumptions used by previous works mentioned above.

Original:  M ETHOD
Revised:  M ETHOD

Original: In this section, we first describe our proposed variational constraint and then present the overall network architecture leading to the overall loss function.
Revised: In this section, we first describe our proposed variational constraint and then present the overall network architecture leading to the overall loss function.

Original: 3. V ARIATIONAL C ONSTRAINT
Revised: 3. V ARIATIONAL C ONSTRAINT

Original: In this part, we introduce the variational constraint and how it can be useful for depth estimation.
Revised: In this part, we introduce the variational constraint and how it can be useful for depth estimation.

Original: Consider the unscaled depth map as Z u ∈ R H × W , with ( H, W ) symbolizing the height and width, respectively.
Revised: Consider the unscaled depth map as Z u ∈ R H × W , with ( H, W ) symbolizing the height and width, respectively.

Original: Assuming Γ x ∈ R H × W and Γ y ∈ R H × W as the gradient of Z u in the x and y axis, we write
Revised: Assuming Γ x ∈ R H × W and Γ y ∈ R H × W as the gradient of Z u in the x and y axis, we write

Original: [Equation]
Revised: [Equation]

Original: Here, x and y subscript corresponds to the direction from left to right ( x -axis) and top to bottom of the image ( y -axis), respectively.
Revised: Here, x and y subscript corresponds to the direction from left to right ( x -axis) and top to bottom of the image ( y -axis), respectively.

Original: Elaborating on this, we can write
Revised: Elaborating on this, we can write

Original: [Equation]
Revised: [Equation]

Original: Suppose we augment Eq.(2) expression for all ( i, j ) , i ∈ { 1 , ..., H } and j ∈ { 1 , ..., W } .
Revised: Suppose we augment Eq.(2) expression for all ( i, j ) , i ∈ { 1 , ..., H } and j ∈ { 1 , ..., W } .

Original: In that case, we will end up with an over-determined system with 2 HW equations in total.
Revised: In that case, we will end up with an over-determined system with 2 HW equations in total.

Original: Given the predicted Γ x and Γ y , we aim to recover the HW unknown variables in Z u .
Revised: Given the predicted Γ x and Γ y , we aim to recover the HW unknown variables in Z u .

Original: However, some of the equations could be spurious and deteriorate the overall depth estimation result rather than improving it.
Revised: However, some of the equations could be spurious and deteriorate the overall depth estimation result rather than improving it.

Original: As a result, we must be mindful about selecting the equation that respects the imposed first-order constraint and maintains the depth gradient to have a meaningful fitting for better generalization.
Revised: As a result, we must be mindful about selecting the equation that respects the imposed first-order constraint and maintains the depth gradient to have a meaningful fitting for better generalization.

Original: To that end, we introduce confidence weight Σ x ∈ [0 , 1] H × W , Σ y ∈ [0 , 1] H × W for gradient along x, y direction.
Revised: To that end, we introduce confidence weight Σ x ∈ [0 , 1] H × W , Σ y ∈ [0 , 1] H × W for gradient along x, y direction.

Original: Consequently, we multiply the above two equations by the confidence weight term Σ
Revised: Consequently, we multiply the above two equations by the confidence weight term Σ

Original: i,jx and Σ i,j y , respectively.
Revised: i,jx and Σ i,jy , respectively.

Original: On one hand, if the confidence is close to 1, the equation will have priority to be satisfied by the optimal Z u .
Revised: On one hand, if the confidence is close to 1, the equation will have priority to be satisfied by the optimal Z u .

Original: On the other hand, if the confidence is close to 0, we must ignore the equation.
Revised: On the other hand, if the confidence is close to 0, we must ignore the equation.

Original: For better understanding, we illustrate the first order difference and weighted matrix construction in Fig.3 (a) and Fig.3(b).
Revised: For better understanding, we illustrate the first order difference and weighted matrix construction in Fig.2 (a) and Fig.2(b).

Original: Next, we reshape the Σ x , Σ y , Γ x , Γ y , and Z u into column vectors ˜Σ x ∈ [0 , 1] HW × 1 , ˜Σ y ∈ [0 , 1] HW × 1 , ˜Γ x ∈ R HW × 1 , ˜Γ y ∈ R HW × 1 , and ˜ Z u ∈ R HW × 1 , respectively.
Revised: Next, we reshape the Σ x , Σ y , Γ x , Γ y , and Z u into column vectors ˜Σ x ∈ [0 , 1] HW × 1 , ˜Σ y ∈ [0 , 1] HW × 1 , ˜Γ x ∈ R HW × 1 , ˜Γ y ∈ R HW × 1 , and ˜ Z u ∈ R HW × 1 , respectively.

Original: Organizing ˜Σ = diag ([˜Σ x ; ˜Σ y ])
Revised: Organizing ˜Σ = diag ([˜Σ x ; ˜Σ y ])

Original: ∈ R 2 HW × 2 HW and ˜Γ = concat [˜Γ x ; ˜Γ y
Revised: ∈ R 2 HW × 2 HW and ˜Γ = concat [˜Γ x ; ˜Γ y

Original: ] ∈ R 2 HW × 1 , we can write the overall expression in a compact matrix form using simple algebra as follows
Revised: ] ∈ R 2 HW × 1 , we can write the overall expression in a compact matrix form using simple algebra as follows

Original: [Equation]
Revised: [Equation]

Original: where P ∈ { 1 , − 1 } 2 HW × HW is the first-order difference operator.

Original: Specifically, P is a sparse matrix with only a few elements as 1 or -1.

Original: The i th row of P provides the first-order difference operator for the i th equation. The position of 1 and -1 indicates which pair of neighbors to be considered for the constraint. Fig.3 (b) provides a visual intuition about this matrix equation. Eq.(3) can be utilized to recover Z u from the predicted Γ x , Γ y , Σ x , and Σ y .
Revised: ( b ) VL ay er ( c ) U p s a m p

Original: As alluded to above, we have more equations than unknowns, hence, we resort to recovering the optimal depth map

Original: [Figure] ) V - L ay [Figure] ( c ) U p s a m p l e a nd R e f i n e
Revised: l e a nd R e f i n e

Original: Upsampling ...
Revised: Upsampling ...

Original: A A
Revised: A A

Original: A reArrange
Revised: A reArrange

Original: Eq.
Revised: Eq.

Original: Feature
Revised: Feature

Original: Depth Depth
Revised: Depth Depth

Original: Refine
Revised: Refine

Original: Depth
Revised: Depth

Original: Feature Feature
Revised: Feature Feature

Original: Refine Refine
Revised: Refine Refine

Original: A
Revised: A

Original: Depth
Revised: Depth

Original: [Figure]
Revised: [Figure]

Original: Difference Vector
Revised: DifferenceOperator where P ∈ { 1 , − 1 } 2 HW × HW is the first-order difference operator. Specifically, P is a sparse matrix with only a few elements as 1 or -1. The i th row of P provides the first-order difference operator for the i th equation.

Original: DifferenceOperator
Revised: The position of 1 and -1 indicates which pair of neighbors to be considered for the constraint. Fig.2 (b) provides a visual intuition about this matrix equation. Eq.(3) can be utilized to recover Z u from the predicted Γ x , Γ y , Σ x , and Σ y . As alluded to above, we have more equations than unknowns, hence, we resort to recovering the optimal depth map ˜

Original: ˜ Z ∗ u ∈ R HW × 1 by minimizing the following equation:
Revised: Z ∗ u ∈ R HW × 1 by minimizing the following equation:

Original: [Equation]
Revised: [Equation]

Original: Refer Appendix for the full derivation.
Revised: Refer Appendix for the full derivation.

Original: The closed-form solution can be written as follows:
Revised: The closed-form solution can be written as follows:

Original: [Equation]
Revised: [Equation]

Original: Denote K ˜Σ ≜ ( P T ˜Σ 2 P ) − 1 P T ˜Σ 2 in Eq.(5), we write overall equation as ˜ Z ∗ u = K ˜Σ ˜Γ .
Revised: Denote K ˜Σ ≜ ( P T ˜Σ 2 P ) − 1 P T ˜Σ 2 in Eq.(5), we write overall equation as ˜ Z ∗ u = K ˜Σ ˜Γ .

Original: Next, we describe the overall network architecture.
Revised: Next, we describe the overall network architecture.

Original: 3. O VERALL N ETWORK A RCHITECTURE
Revised: 3. O VERALL N ETWORK A RCHITECTURE

Original: Our overall network architecture is composed of four main modules as follows.
Revised: Our overall network architecture is composed of four main modules as follows.

Original: (a) Encoder.
Revised: (a) Encoder.

Original: Given an input image, the encoder computes the hierarchical feature maps through a series of stages
Revised: Given an input image, the encoder computes the hierarchical feature maps through a series of stages.

Original: To be precise, our encoder has four stages.
Revised: To be precise, our encoder has four stages.

Original: Each stage contains transformer blocks (Liu et al., 2021b).
Revised: Each stage contains transformer blocks (Liu et al., 2021b).

Original: At the end of each stage, we collect the final feature map as the output of the encoder resulting in the encoded feature maps with strides 4, 8, 16, and 32, respectively.
Revised: At the end of each stage, we collect the final feature map as the output of the encoder resulting in the encoded feature maps with strides 4, 8, 16, and 32, respectively.

Original: Our encoder module is inspired by Liu et al.
Revised: Our encoder module is inspired by Liu et al.

Original: (2021b), a recent state-of-the-art transformer network design.
Revised: (2021b), a recent state-of-the-art transformer network design.

Original: We used it as our backbone by removing the final global pooling layer and fully connected layer.
Revised: We used it as our backbone by removing the final global pooling layer and fully connected layer.

Original: (b) Variational Layer (V-Layer).
Revised: (b) Variational Layer (V-Layer).

Original: The goal of this layer is to compute a map from encoded feature map to unscaled depth map, which adheres to the first order variational constraint.
Revised: The goal of this layer is to compute a map from encoded feature map to unscaled depth map, which adheres to the first order variational constraint.

Original: As of V-layer, we feed the feature maps of strides 16 and 32 as input which is the output of the encoder.
Revised: As of V-layer, we feed the feature maps of strides 16 and 32 as input which is the output of the encoder.

Original: Since these features are at different resolution, we up-sample the feature map of stride 32 to stride 16 via bi-linear interpolation and concatenate to construct I Φ ∈ R C × H × W , where ( H, W, C ) symbolizing the height, width, and the number of channels, respectively.
Revised: Since these features are at different resolution, we up-sample the feature map of stride 32 to stride 16 via bi-linear interpolation and concatenate to construct I Φ ∈ R C × H × W , where ( H, W, C ) symbolizing the height, width, and the number of channels, respectively.

Original: Note that H, W is not the same as the original resolution of the ground-truth depth and images.
Revised: Note that H, W is not the same as the original resolution of the ground-truth depth and images.

Original: We use of two convolutional layers on I Φ to predict the depth gradient and corresponding weight for each pixel as follows:
Revised: We use of two convolutional layers on I Φ to predict the depth gradient and corresponding weight for each pixel as follows:

Original: [Equation]
Revised: [Equation]

Original: where, f ( I Φ ; θ ) denotes the convolutional layers with parameters θ .
Revised: where, f ( I Φ ; θ ) denotes the convolutional layers with parameters θ .

Original: The predicted depth gradients Γ x and Γ y are observed to be more accurate at smooth surface than at boundaries.
Revised: The predicted depth gradients Γ x and Γ y are observed to be more accurate at smooth surface than at boundaries.

Original: This brings us again to the point made above that we must take care of which first order constraint must be included and discarded during regression.
Revised: This brings us again to the point made above that we must take care of which first order constraint must be included and discarded during regression.

Original: Using the Eq.(6) prediction, we construct the variational constraint Eq.(3), and obtain the unscaled depth map following Eq.(5).
Revised: Using the Eq.(6) prediction, we construct the variational constraint Eq.(3), and obtain the unscaled depth map following Eq.(5).

Original: The resulting depth map has a resolution of 1/16 to the original image, which is later upsampled to the appropriate dimension.
Revised: The resulting depth map has a resolution of 1/16 to the original image, which is later upsampled to the appropriate dimension.

Original: To capture more scene features, we generate multiple channels (denoted as S ) of { Γ x , Γ y , Σ x , Σ y } using Eq.(6).
Revised: To capture more scene features, we generate multiple channels (denoted as S ) of { Γ x , Γ y , Σ x , Σ y } using Eq.(6).

Original: As a result, we have a group of depth maps stacked along the channel dimension.
Revised: As a result, we have a group of depth maps stacked along the channel dimension.

Original: For a feature map with spatial resolution H × W , our V-layer has a complexity of O
Revised: For a feature map with spatial resolution H × W , our V-layer has a complexity of O

Original: ( H 3 W 3 ) .
Revised: ( H 3 W 3 ) .

Original: To overcome complexity issue, we perform V-layer operation on feature maps with stride 16 and then up-sample and refine the depth maps in the later stage.
Revised: To overcome complexity issue, we perform V-layer operation on feature maps with stride 16 and then up-sample and refine the depth maps in the later stage.

Original: The V-layer pipeline is shown in Fig.4(b).
Revised: The V-layer pipeline is shown in Fig.3(b).

Original: (c) Upsample and Refine.
Revised: (c) Upsample and Refine.

Original: This module upsamples and refines the input depth map via encoded features at a given depth map resolution.
Revised: This module upsamples and refines the input depth map via encoded features at a given depth map resolution.

Original: To this end, we perform refinement at three different resolutions in a hierarchical manner.
Revised: To this end, we perform refinement at three different resolutions in a hierarchical manner.

Original: Given the V-layer depth map at 1/16 resolution, we first refine the depth via encoded features at this resolution.
Revised: Given the V-layer depth map at 1/16 resolution, we first refine the depth via encoded features at this resolution.

Original: Concretely, this refinement is done using the following set of operations.
Revised: Concretely, this refinement is done using the following set of operations.

Original: (1) concatenate the feature map and the depth map; (2) use one convolutional layer with ReLU activation to fuse the feature and depth information; and (3) predict refined feature and depth map via a convolutional layer.
Revised: (1) concatenate the feature map and the depth map; (2) use one convolutional layer with ReLU activation to fuse the feature and depth information; and (3) predict refined feature and depth map via a convolutional layer.

Original: Later, the refined features and depth map is upsampled and fed into 1/8 for later refinement using the same above set of operations.
Revised: Later, the refined features and depth map is upsampled and fed into 1/8 for later refinement using the same above set of operations.

Original: Finally, the exact is done at 1/ resolution.
Revised: Finally, the exact is done at 1/ resolution.

Original: Note that these steps are performed in a sequel.
Revised: Note that these steps are performed in a sequel.

Original: At the end of this module, we have a depth map of 1/4 of the actual resolution.
Revised: At the end of this module, we have a depth map of 1/4 of the actual resolution.

Original: The upsample and refine procedure is shown in Fig.4(c).
Revised: The upsample and refine procedure is shown in Fig.3(c).

Original: (d) Metric Layer.
Revised: (d) Metric Layer.

Original: We must infer the global scene scale and shift to recover the metric depth.
Revised: We must infer the global scene scale and shift to recover the metric depth.

Original: For this, we perform global max pooling on the encoded feature map of stride 32.
Revised: For this, we perform global max pooling on the encoded feature map of stride 32.

Original: The resulting vector is fed into a stack of fully connected layers to regress the two scalars, i.e., one representing the scale and while other representing the shift.
Revised: The resulting vector is fed into a stack of fully connected layers to regress the two scalars, i.e., one representing the scale and while other representing the shift.

Original: Using the feature map of stride 32 is motivated by the observation that we have a much richer global scene context using it than at higher depth resolution.
Revised: Using the feature map of stride 32 is motivated by the observation that we have a much richer global scene context using it than at higher depth resolution.

Original: It also provides a good compromise between computational complexity and accuracy.
Revised: It also provides a good compromise between computational complexity and accuracy.

Original: 3. L OSS F UNCTION
Revised: 3. L OSS F UNCTION

Original: Depth Loss.
Revised: Depth Loss.

Original: It estimates the scale-invariant difference between the ground-truth depth and prediction at train time (Eigen et al., 2014).
Revised: It estimates the scale-invariant difference between the ground-truth depth and prediction at train time (Eigen et al., 2014).

Original: The difference is computed by upsampling the predicted depth map to the same resolution as the ground truth via bi-linear interpolation.
Revised: The difference is computed by upsampling the predicted depth map to the same resolution as the ground truth via bi-linear interpolation.

Original: Denoting the predicted and ground-truth depth as ˆ Z ∈ R m × n , Z gt ∈ R m × n we compute the depth loss as follows 
Revised: Denoting the predicted and ground-truth depth as ˆ Z ∈ R m × n , Z gt ∈ R m × n we compute the depth loss as follows [Equation] , where , e i,j = log

Original: [Equation]
Revised: [Equation]

Original:  Here, N is the number of positions with valid measurements and α ∈ [0 , 1] is a hyper-parameter.
Revised: Z [Equation] Here, N is the number of positions with valid measurements and α ∈ [0 , 1] is a hyper-parameter.

Original: Note that the above loss is used for valid measurements only.
Revised: Note that the above loss is used for valid measurements only.

Original: Variational Loss.
Revised: Variational Loss.

Original: We define this loss using the output of V-layer.
Revised: We define this loss using the output of V-layer.

Original: Suppose the ground-truth depth map to be Z gt ∈
Revised: Suppose the ground-truth depth map to be Z gt ∈

Original: R m × n and the predicted depth map for S channel as Z u ∈ R S × H × W .
Revised: R m × n and the predicted depth map for S channel as Z u ∈ R S × H × W .

Original: Since the depth resolution is not same at this layer, we down-sample the ground-truth.
Revised: Since the depth resolution is not same at this layer, we down-sample the ground-truth.

Original: It is observed via empirical study that low-resolution depth map in fact help capture the first-order variational loss among distant neighbors.
Revised: It is observed via empirical study that low-resolution depth map in fact help capture the first-order variational loss among distant neighbors.

Original: Accordingly, we down sample the Z gt instead of upsamping Z u .
Revised: Accordingly, we down sample the Z gt instead of upsamping Z u .

Original: We down-sampled Z gt denoted as Q gt ∈ R H × W by random pooling operation, i.e., we randomly select a location where we have a valid measurement since ground-truth data may have pixels with no depth values.
Revised: We down-sampled Z gt denoted as Q gt ∈ R H × W by random pooling operation, i.e., we randomly select a location where we have a valid measurement since ground-truth data may have pixels with no depth values.

Original: The coordinates of selected location in Z gt (cid:55)→ Z u ∈ R S × H × W and the corresponding depth value is put in ˆ Q ∈ R S × H × W via bi-linear interpolation.
Revised: The coordinates of selected location in Z gt (cid:55)→ Z u ∈ R S × H × W and the corresponding depth value is put in ˆ Q ∈ R S × H × W via bi-linear interpolation.

Original: We compute the variational loss as
Revised: We compute the variational loss as

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original:  where N ′ is the number of positions having valid measurements, ∇ symbolises the first-order difference operator, and Conv refers to the convolutional layer.
Revised: [Table] where N ′ is the number of positions having valid measurements [Table] ∇ symbolises the first-order difference operator

Revised: [Table]

Original: Total Loss.
Revised: and Conv refers to the convolutional layer. Here [Table] we use the Conv layer to fuse S depth maps into a single depth map and also compute its horizontal and vertical gradient.

Original:  We define the total loss as the sum of the depth loss and the variational loss i.e.,
Revised: Total Loss. We define the total loss as the sum of the depth loss and the variational loss i.e.,

Original: L = L depth + λ L var , where λ is the regularization parameter set to 0.1 for all our experiments.[Table] [Table]
Revised: L = L depth + λ L var , where λ is the regularization parameter set to 0.1 for all our experiments.

Original:  E XPERIMENTS AND R ESULTS
Revised:  E XPERIMENTS AND R ESULTS

Original: Implementation Details We implemented our method in PyTorch 1.7.1 (Python 3.8) with CUDA 11.0.
Revised: Implementation Details We implemented our method in PyTorch 1.7.1 (Python 3.8) with CUDA 11.0.

Original: The software issimulated and evaluated on a computing machine with Quadro-RTXGPU (24GB Memory).
Revised: The software is evaluated on a computing machine with Quadro-RTX-6000 GPU.

Original: Datasets.
Revised: Datasets.

Original: We evaluated our approach on threepopular benchmark dataset namely NYU Depth V (Silberman et al., 2012), KITTI (Geiger et al., 2012) [Table] and SUN RGB-D (Song et al., 2015).
Revised: We performed experiments on three benchmark dataset namely NYU Depth V2 (Silberman et al., 2012), KITTI (Geiger et al., 2012), and SUN RGB-D (Song et al., 2015).

Original: (a) 
Revised: (a) NYU Depth

Original: NYU Depth V2 contains images with 480 × 640 resolution with depth values ranging from 0 to 10 meters.
Revised: V2 contains images with 480 × 640 resolution with depth values ranging from 0 to 10 meters.

Original: We follow the train and test set split from Lee et al.
Revised: We follow the train and test set split from Lee et al.

Original: [Table] which contains 24, train images and 654 test images.
Revised: (2019), which contains 24,231 train images andtest images.

Original: (b) KITTI contains images with 352 × 1216 resolution where depth values ranges 0 to 80 meters.
Revised: (b) KITTI contains images with 352 × 1216 resolution where depth values ranges 0 to 80 meters.

Original: The official split provides 42,949 training, 1,000 validation, and 500 test images.
Revised: The official split provides 42,949 training, 1,000 validation, and 500 test images.

Original: Eigen et al.
Revised: Eigen et al.

Original: (2014) provide another train and test set split for this dataset which has 23,488 training and 697 test images.
Revised: (2014) provide another train and test set split for this dataset which has 23,488 training andtest images.

Original: (c) SUN
Revised: (c) SUN

Original: RGB-D We preprocess its images to 480 ×resolution for consistency.
Revised: RGB-D We preprocess its images to 480 × 640 resolution for consistency.

Original: The depth values ranges from 0 to 10 meters.
Revised: The depth values ranges from 0 to 10 meters.

Original: We use the official test set (5050 images) for evaluation.
Revised: We use the official test set (5050 images) for evaluation.

Original: Training Details.
Revised: Training Details.

Original: We use (Liu et al., 2021b) network as our backbone, which is pre-trained on
Revised: We use (Liu et al., 2021b) network as our backbone, which is pre-trained on

Original: ImageNet (Deng et al., 2009).
Revised: ImageNet (Deng et al., 2009).

Original: We use the Adam optimizer (Kingma & Ba, 2014) without weight decay.
Revised: We use the Adam optimizer (Kingma & Ba, 2014) without weight decay.

Original: We decrease the learning rate from 3 e − 5 to 1 e − 5 by the cosine annealing scheduler.
Revised: We decrease the learning rate from 3 e − 5 to 1 e − 5 by the cosine annealing scheduler.

Original: To avoid over-fitting, we augment the images by horizontal flipping.
Revised: To avoid over-fitting, we augment the images by horizontal flipping.

Original: For KITTI (Geiger et al., 2012), the model is trained for 10 epochs for the official split and 20 epochs for the Eigen split (Eigen et al., 2014).
Revised: For KITTI (Geiger et al., 2012), the model is trained for 10 epochs for the official split and 20 epochs for the Eigen split (Eigen et al., 2014).

Original: For NYU Depth V2 (Silberman et al., 2012), the model is trained for 20 epochs.
Revised: For NYU Depth V2 (Silberman et al., 2012), the model is trained for 20 epochs.

Original: Evaluation Metrics.
Revised: Evaluation Metrics.

Original: We report statistical results on popular evaluation metrics such as square root of the Scale Invariant Logarithmic error ( SILog ), Relative Squared error ( Sq Rel ), Relative Absolute
Revised: We report statistical results on popular evaluation metrics such as square root of the Scale Invariant Logarithmic error ( SILog ), Relative Squared error ( Sq Rel ), Relative Absolute

Original: Error ( Abs Rel ), Root Mean Squared error ( RMS ), and threshold accuracy.
Revised: Error ( Abs Rel ), Root Mean Squared error ( RMS ), and threshold accuracy.

Original: Mathematical definition related to each one of them is providedin the Appendix. 
Revised: Mathematical definition related to each one of them is provided Appendix. [Table] [Table]

Original: 4. C OMPARISON TO THE S TATE - OF - THE -A RT
Revised: 4. C OMPARISON TO THE S TATE - OF - THE -A RT

Original: Tab.(1), Tab.(2), Tab.(3), and Tab.(4) provide statistical comparison results with the competing methods on NYU Depth V2, KITTI official split, KITTI Eigen split, and SUN RGB-D, respectively.Our [Table] [Table]
Revised: Tab.(1), Tab.(2), Tab.(3), and Tab.(4) provide statistical comparison results with the competing methods on NYU Depth V2, KITTI official split, KITTI Eigen split, and SUN RGB-D, respectively.

Original:  proposed approach shows the best results for all the evaluation metrics.
Revised: Our proposed approach shows the best results for all the evaluation metrics.

Original: Particularly on the NYU test set, we reduce the SILog error from the previous best result, 9.102 to 8.439, and increase δfrom 0.922 to 0.937.
Revised: Particularly on the NYU test set, we reduce the SILog error from the previous best result, 9.102 to 8.439, and increase δ 1 from 0.922 to 0.937.

Original: Visualization of predicted depth maps and intermediate results in V-layer  are presented in the Appendix.
Revised: More qualitative results including V-layer output are presented in the Appendix.

Original: For the SUN RGB-D test set, all competing models, including ours, are trained on the NYU Depth
Revised: For the SUN RGB-D test set, all competing models, including ours, are trained on the NYU Depth

Original: V2 training set (Silberman et al., 2012) without fine-tuning on the SUN RGB-D.
Revised: V2 training set (Silberman et al., 2012) without fine-tuning on the SUN RGB-D.

Original: In addition, we align the predictions from all the models with the ground truth by a scale and shift following Ranftl et al.
Revised: In addition, we align the predictions from all the models with the ground truth by a scale and shift following Ranftl et al.

Original: Tab.(4) results show our method’s better generalization capability than other approaches.
Revised: Tab.(4) results show our method’s better generalization capability than other approaches.

Original: Extensive visual results are provided in the Appendix and supplementary video.
Revised: Extensive visual results are provided in the Appendix and supplementary video.

Original: 4. A BLATION S TUDY
Revised: 4. A BLATION S TUDY

Original: All the ablation presented below is conducted on NYU Depth V2 test set (Silberman et al., 2012).
Revised: All the ablation presented below is conducted on NYU Depth V2 test set (Silberman et al., 2012).

Original: (i) Effect of V-Layer.
Revised: (i) Effect of V-Layer.

Original: To understand the benefit and outcome of our introduced variational layer compared to other popular alternative layers in deep-neural networks for this problem, we performed this ablation study.
Revised: To understand the benefit and outcome of our introduced variational layer compared to other popular alternative layers in deep-neural networks for this problem, we performed this ablation study.

Original: We replace our V-layer firstly with a convolutional layer and later with a selfattention layer.
Revised: We replace our V-layer firstly with a convolutional layer and later with a selfattention layer.

Original: Tab.(5) provides the depth prediction accuracy for this ablation.
Revised: Tab.(5) provides the depth prediction accuracy for this ablation.

Original: For each introduced layer in Tab.(5), the first and second rows show the performance of the depth map predicted with (w) and without (w/o) subsequent refinements (refer Sec.3.2(c)), respectively.
Revised: For each introduced layer in Tab.(5), the first and second rows show the performance of the depth map predicted with (w) and without (w/o) subsequent refinements (refer Sec.3.2(c)), respectively.

Original: For the self-attention layer, we follow the ViT (Dosovitskiy et al., 2020) and set the patch size to be one as we use the feature map with stride 16.
Revised: For the self-attention layer, we follow the ViT (Dosovitskiy et al., 2020) and set the patch size to be one as we use the feature map with stride 16.

Original: We also adopt the learnable position embedding (PE) with 128dimension.
Revised: We also adopt the learnable position embedding (PE) with 128dimension.

Original: We set the number of heads to be 4 and the number of hidden units to be 512.
Revised: We set the number of heads to be 4 and the number of hidden units to be 512.

Original: As shown in Tab.(5), our V-layer indeed help improve the accuracy of depth prediction compared to other well-known layers.
Revised: As shown in Tab.(5), our V-layer indeed help improve the accuracy of depth prediction compared to other well-known layers.

Original: More experiments on KITTI and SUN RBG-D are provided in the Appendix.
Revised: More experiments on KITTI and SUN RBG-D are provided in the Appendix.

Original: (ii) Performance with Different Network Backbone.
Revised: (ii) Performance with Different Network Backbone.

Original: We evaluate the effects of our V-layer with different types of network backbones.
Revised: We evaluate the effects of our V-layer with different types of network backbones.

Original: For this ablation, we use Swin-Large (Liu et al., 2021b),
Revised: For this ablation, we use Swin-Large (Liu et al., 2021b),

Original: [Table]
Revised: [Table]

Revised: 0.990 w/ 8.

Original:  Swin-Small (Liu et al., 2021b), and ConvNeXt-Small (Liu et al., 2022).
Revised: V-Layer w/o 8. 0.990 w/ 8. [Table] Swin-Small (Liu et al., 2021b), and ConvNeXt-Small (Liu et al., 2022).

Original: The SILog error is shown in Fig.5.
Revised: The SILog error is shown in Fig.4.

Original: The results show that our V-layer improves the transformer and the convolutional network performance.
Revised: The results show that our V-layer improves the transformer and the convolutional network performance.

Original: An important observation is that our V-layer shows excellent improvements in depth prediction accuracy on weaker network backbones.
Revised: An important observation is that our V-layer shows excellent improvements in depth prediction accuracy on weaker network backbones.

Original: (iii) Performance with change in the value of S .
Revised: (iii) Performance with change in the value of S .

Original: For this ablation, we change the value of S in the
Revised: For this ablation, we change the value of S in the

Original: V-layer and observed its effects (refer Sec.3.2).
Revised: V-layer and observed its effects (refer Sec.3.2).

Original: By increasing S , we generate more channels of ˜Γ and ˜Σ which in-effect increases V-layer parameters.
Revised: By increasing S , we generate more channels of ˜Γ and ˜Σ which in-effect increases V-layer parameters.

Original: In the subsequent step, we expand the number of channels to 128 by a convolutional layer to use the subsequent layer as it is.
Revised: In the subsequent step, we expand the number of channels to 128 by a convolutional layer to use the subsequent layer as it is.

Original: The results are shown in Tab.(6).
Revised: The results are shown in Tab.(6).

Original: For reference, we also present the result by replacing the V-layer with a convolutional layer in the first row in Tab.(6).
Revised: For reference, we also present the result by replacing the V-layer with a convolutional layer in the first row in Tab.(6).

Original: By increasing the number of channels, we can reduce the SILog error, at the price of the speed (FPS).
Revised: By increasing the number of channels, we can reduce the SILog error, at the price of the speed (FPS).

Original: Yet, no real benefit is observed with S more than 16. 
Revised: Yet, no real benefit is observed with S more than 16. [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: (iv) Effect of Confidence weight matrix & Difference Operator in V-Layer.
Revised: (iv) Effect of Confidence weight matrix & Difference Operator in V-Layer.

Original: For this ablation, we study the our network’s depth prediction under four different settings.
Revised: For this ablation, we study the our network’s depth prediction under four different settings.

Original: (a) without Vlayer and replace it with convolutional layer (b) without the confidence weight matrix (c) with learnable difference operator and (d) our full model.
Revised: (a) without Vlayer and replace it with convolutional layer (b) without the confidence weight matrix (c) with learnable difference operator and (d) our full model.

Original: The depth prediction accuracy observed under these settings is provided in Tab.(7).
Revised: The depth prediction accuracy observed under these settings is provided in Tab.(7).

Original: Clearly, our full model has better accuracy.
Revised: Clearly, our full model has better accuracy.

Original: An important empirical observation, we made during this test is when we keep P learnable V-layer has more learnable parameters, the performance becomes worse than with fixed difference operator.
Revised: An important empirical observation, we made during this test is when we keep P learnable V-layer has more learnable parameters, the performance becomes worse than with fixed difference operator.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 4. N ETWORK PROCESSING TIME & P ARAMETERS
Revised: 4. N ETWORK PROCESSING TIME & P ARAMETERS

Original: We compared our method’s inference time and the number of model parameters to the AdaBins (Bhat et al., 2021) and the NeWCRFs (Yuan et al., 2022).
Revised: We compared our method’s inference time and the number of model parameters to the AdaBins (Bhat et al., 2021) and the NeWCRFs (Yuan et al., 2022).

Original: The inference time is measured on the NYU Depth V2 test set with batch size 1.
Revised: The inference time is measured on the NYU Depth V2 test set with batch size 1.

Original: We have removed the ensemble tricks in AdaBins and NeWCRFs. As is shown in Tab.(8) [Table]
Revised: We have removed the ensemble tricks in AdaBins and NeWCRFs for an unbiased evaluation, resulting in a slight increase in SILog error as compared to Tab.(1) statistics.

Original:  our method is faster and better than AdaBins and NeWCRFs using Swin-Small backbone.
Revised: As is shown in Tab.(8), our method is faster and better than AdaBins and NeWCRFs using Swin-Small backbone.

Original: With the same backbone as the NeWCRFs, i
Revised: With the same backbone as the NeWCRFs, i

Original: [Table]
Revised: [Table]

Original: e. Swin-Large, we achieve much better depth prediction results.
Revised: e., Swin-Large, we achieve much better depth prediction results.

Original: Hence, our method with Swin-Small backbone provides a better balance between accuracy, speed and memory foot-print.
Revised: Hence, our method with Swin-Small backbone provides a better balance between accuracy, speed and memory foot-print

Original: [Table] 
Revised: [Table] [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In conclusion, a simple and effective approach for inferring scene depth from a single image is introduced.
Revised: In conclusion, a simple and effective approach for inferring scene depth from a single image is introduced.

Original: The proposed SIDP approach is shown to better exploit the rigid scene prior, which is generally overlooked by the existing neural network-based methods.
Revised: The proposed SIDP approach is shown to better exploit the rigid scene prior, which is generally overlooked by the existing neural network-based methods.

Original: Our approach does not make explicit assumptions about the scene other than the scene gradient regularity, which holds for typical indoor or outdoor scenes.
Revised: Our approach does not make explicit assumptions about the scene other than the scene gradient regularity, which holds for typical indoor or outdoor scenes.

Original: When tested on popular benchmark datasets, our method shows significantly better results than the prior art, both qualitatively and quantitatively.
Revised: When tested on popular benchmark datasets, our method shows significantly better results than the prior art, both qualitatively and quantitatively.

Original: A A PPENDIX A. T RAINING D ETAILS
Revised: A A PPENDIX A. T RAINING D ETAILS

Original: We implement our framework in PyTorch (Paszke et al., 2019).
Revised: We implement our framework in PyTorch (Paszke et al., 2019).

Original: We adopt the Swin-Large (Liu et al., 2021b) as our backbone to conduct ablation experiments and compare with the state-of-the-arts.
Revised: We adopt the Swin-Large (Liu et al., 2021b) as our backbone to conduct ablation experiments and compare with the state-of-the-arts.

Original: And the backbone is pre-trained on ImageNet-22K (Deng et al., 2009) by image classification.
Revised: And the backbone is pre-trained on ImageNet-22K (Deng et al., 2009) by image classification.

Original: For training, we use the Adam optimizer (Kingma & Ba, 2014) without weight decay.
Revised: For training, we use the Adam optimizer (Kingma & Ba, 2014) without weight decay.

Original: We decrease the learning rate from 3e-5 to 1e-5 by the cosine annealing scheduler.
Revised: We decrease the learning rate from 3e-5 to 1e-5 by the cosine annealing scheduler.

Original: To avoid over-fitting, we augment the images by horizontal flipping.
Revised: To avoid over-fitting, we augment the images by horizontal flipping.

Original: For both training and test, we keep the resolution of images to be 352 × 1216 in KITTI (Geiger et al., 2012), and 480 × 640 in both NYU Depth V2 (Silberman et al., 2012) and SUN RGB-D (Song et al., 2015).
Revised: For both training and test, we keep the resolution of images to be 352 × 1216 in KITTI (Geiger et al., 2012), and 480 × 640 in both NYU Depth V2 (Silberman et al., 2012) and SUN RGB-D (Song et al., 2015).

Original: In KITTI, the model is trained for 10 epochs for the official split and 20 epochs for the Eigen split (Eigen et al., 2014).
Revised: In KITTI, the model is trained for 10 epochs for the official split and 20 epochs for the Eigen split (Eigen et al., 2014).

Original: In NYU Depth V2, the model is trained for 20 epochs.
Revised: In NYU Depth V2, the model is trained for 20 epochs.

Original: We set the batch size to be 4 and 8 respectively for ablation experiments and comparison with the state-of-the-art.
Revised: We set the batch size to be 4 and 8 respectively for ablation experiments and comparison with the state-of-the-art.

Original: A. E VALUATION M ETRICS
Revised: A. E VALUATION M ETRICS

Original: Supposing the predicted and ground-truth depth to be ˆ Z ∈ R m × n and Z gt ∈
Revised: Supposing the predicted and ground-truth depth to be ˆ Z ∈ R m × n and Z gt ∈

Original: R m × n , respectively, and the number of valid pixels to be N .
Revised: R m × n , respectively, and the number of valid pixels to be N .

Original: We follow the existing methods (Yuan et al., 2022) and utilize the following measures for quantitative evaluation:
Revised: We follow the existing methods (Yuan et al., 2022) and utilize the following measures for quantitative evaluation:

Original: • square root of the Scale Invariant Logarithmic error ( SILog ): 1 N (cid:80) i,j ( e i,j ) 2 − 1 N 2 ( (cid:80) i,j e i,j ) 2 , where e i,j = log ˆ
Revised: • square root of the Scale Invariant Logarithmic error ( SILog ): 1 N (cid:80) i,j ( e i,j ) 2 − 1 N 2 ( (cid:80) i,j e i,j ) 2 , where e i,j = log ˆ

Original: Z i,j − log Z i,jgt ; • Relative Squared error ( Sq Rel ): 1 N (cid:80) i,j
Revised: Z i,j − log Z i,jgt ; • Relative Squared error ( Sq Rel ): 1 N (cid:80) i,j

Original: ( ˆ Z i,j − Z i,jgt ) 2 /Z
Revised: ( ˆ Z i,j − Z i,jgt ) 2 /Z

Original: i,jgt ; • Relative Absolute Error ( Abs Rel ): 1 N (cid:80) i,j | ˆ Z i,j − Z i,jgt | /Z
Revised: i,jgt ; • Relative Absolute Error ( Abs Rel ): 1 N (cid:80) i,j | ˆ Z i,j − Z i,jgt | /Z

Original: i,jgt ; • Root Mean Squared error ( RMS ): 1 N (cid:113)(cid:80) i,j
Revised: i,jgt ; • Root Mean Squared error ( RMS ): 1 N (cid:113)(cid:80) i,j

Original: ( ˆ Z i,j − Z i,jgt ) 2 ; • Root Mean Squared Logarithmic error ( RMS log ): 1 N (cid:113)(cid:80) i,j ( e i,j ) 2 ; • threshold accuracy ( δ k ): percentageof ˆ Z i,j s t. max( ˆ Z i,j Z i,jgt , Z i,jgt ˆ Z i,j ) < 1 .
Revised: ( ˆ Z i,j − Z i,jgt ) 2 ; • Root Mean Squared Logarithmic error ( RMS log ): 1 N (cid:113)(cid:80) i,j ( e i,j ) 2 ; • threshold accuracy ( δ k ): percentage

Original: 25 k .
Revised: [Table]

Original: A. D ERIVATION OF THE V ARIATIONAL C ONSTRAINT
Revised: A. D ERIVATION OF THE V ARIATIONAL C ONSTRAINT

Original: In this part, we introduce the derivation of the variational constraint in more detail.
Revised: In this part, we introduce the derivation of the variational constraint in more detail.

Original: Firstly, we present the specific form of the difference operator P , the confidence weight matrix ˜Σ , the difference vector ˜Γ and the depth vector ˜ Z u , when the depth map Z u is at resolution 2 × 2 .
Revised: Firstly, we present the specific form of the difference operator P , the confidence weight matrix ˜Σ , the difference vector ˜Γ and the depth vector ˜ Z u , when the depth map Z u is at resolution 2 × 2 .

Original: Secondly we show the derivation of the optimal solution ˜
Revised: Secondly we show the derivation of the optimal solution ˜

Original: Z ∗ u .
Revised: Z ∗ u .

Original: Overall Matrix Form Suppose the unscaled depth map to be Z u ∈ R 2 × 2 , the difference map to be Γ x ∈ R 2 × 2 and Γ y ∈ R 2 × 2 , and the corresponding confidence weight to be Σ x ∈ [0 , 1] 2 × 2 and Σ y ∈ [0 , 1] 2 × 2 .
Revised: Overall Matrix Form Suppose the unscaled depth map to be Z u ∈ R 2 × 2 , the difference map to be Γ x ∈ R 2 × 2 and Γ y ∈ R 2 × 2 , and the corresponding confidence weight to be Σ x ∈ [0 , 1] 2 × 2 and Σ y ∈ [0 , 1] 2 × 2 .

Original: We first reorganize the elements in Γ x , Γ y , Σ x , Σ y to construct the difference vector ˜Γ ∈ R 8 × 1 and the confidence weight matrix ˜Σ ∈ R 8 × 8 :
Revised: We first reorganize the elements in Γ x , Γ y , Σ x , Σ y to construct the difference vector ˜Γ ∈ R 8 × 1 and the confidence weight matrix ˜Σ ∈ R 8 × 8 :

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: Σ
Revised: Γ

Revised: [Equation]

Revised: Γ

Original: [Equation] 
Revised: [Equation] 

Original:         Σ 2 , 1 x        Σ 2 , 2 x        Σ 1 , 1 y        Σ 1 , 2 y        Σ 2 , 1 y        Σ 2 , 2 y     .
Revised:   , ˜Σ =    Σ 1 , 1 x        Σ 1 , 2 x        Σ 2 , 1 x        Σ 2 , 2 x        Σ 1 , 1 y        Σ 1 , 2 y        Σ 2 , 1 y        Σ 2 , 2 y    .

Original: Next, we apply the difference operator P ∈ R 8 × 4 :
Revised: Next, we apply the difference operator P ∈ R 8 × 4 :

Original: P = 
Revised: P = 

Original:  −    −− 1    −    −            on the depth vector ˜ Z u ∈ R 4 × 1 : [Equation] u
Revised:  −    −

Original: [Equation]
Revised: [Equation]

Original:  Z 1 , 1 u [Equation] [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  u [Equation] [Equation] u
Revised:   on the depth vector ˜ Z u ∈ R 4 × 1 :

Original: [Equation]
Revised: [Equation]

Original: Finally, we construct the following constraint equation:
Revised: Finally, we construct the following constraint equation:

Original: [Equation]
Revised: [Equation]

Original: In the above equation, each row represents a weighted constraint for the first-order difference along the x -axis:Σ
Revised: In the above equation, each row represents a weighted constraint for the first-order difference along the x -axis:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: or along the y -axis:
Revised: or along the y -axis:

Original: [Equation]
Revised: [Equation]

Original: where ( Z i,j +1 u − Z i,ju )
Revised: where ( Z i,j +1 u − Z i,ju )

Original: =
Revised: =

Original: Γ i,jx and ( Z i +1 ,j u − Z i,ju ) =
Revised: Γ i,jx and ( Z i +1 ,j u − Z i,ju ) =

Original: Γ i,jy are the variational constraints for the first-order difference, while Σ i,jx and Σ i,jy are the confidence weights for the constraints.
Revised: Γ i,jy are the variational constraints for the first-order difference, while Σ i,jx and Σ i,jy are the confidence weights for the constraints.

Original: Optimal Solution Given ˜Σ and ˜Γ , we search for the optimal depth vector ˜
Revised: Optimal Solution Given ˜Σ and ˜Γ , we search for the optimal depth vector ˜

Original: Z ∗ u by minimizing the residual of Eq.(9): 
Revised: Z ∗ u by minimizing the residual of Eq.(9): [Equation]

Original: [Equation]
Revised: [Equation]

Original: Firstly, the objective function, || ˜Σ(
Revised: Firstly, the objective function, || ˜Σ(

Original: P ˜
Revised: P ˜

Original: Z u − ˜Γ) || 2 , is convex with respect to ˜ Z u .
Revised: Z u − ˜Γ) || 2 , is convex with respect to ˜ Z u .

Original: Because || · || 2 is a convex function.
Revised: Because || · || 2 is a convex function.

Original: A composition of || · || 2 and affine function , ˜Σ( P ˜
Revised: A composition of || · || 2 and affine function , ˜Σ( P ˜

Original: Z
Revised: Z

Original: u − ˜Γ) , is still convex.
Revised: u − ˜Γ) , is still convex.

Original: Secondly, the optimal solution of a convex function can be found at where the first derivative is zero.
Revised: Secondly, the optimal solution of a convex function can be found at where the first derivative is zero.

Original: Thereby we obtain the final solution ˜
Revised: Thereby we obtain the final solution ˜

Original: Z ∗ u =
Revised: Z ∗ u =

Original: ( P T ˜Σ 2 P ) − 1 P T ˜Σ 2 ˜Γ .
Revised: ( P T ˜Σ 2 P ) − 1 P T ˜Σ 2 ˜Γ .

Original: A. A BLATION ON KITTI AND SUN RGB-D
Revised: A. N ETWORK S TRUCTURE D ETAILS

Revised: In this part, we introduce the detailed structure of our framework. In general we set the kernel size of convolution layers to be 3 unless otherwise stated. (a) Encoder

Revised: We adopt the Swin-Large (Liu et al., 2021b) as our backbone.

Revised: The network first divides the image into patches each of size 4 × 4 , and embeds each patch into a 96 -dimensional vector.

Revised: The above procedure is implemented by a convolution layer with kernel size 4 and stride 4.

Revised: Then there are 4 transformation stages to be applied to the embedded vector.

Revised: The first, second, third and forth stages include 2 , 2 , 18 , 2 blocks, respectively.

Revised: More specifically, each block will divide the feature map into non-overlapped windows of size 12 × 12 , and compute new features within each window following the transformer (Vaswani et al., 2017).

Revised: The feature channels in the 4 stages are 192, 384, 768 and 1536, respectively, and the number of heads are 6, 12, 24, and 48, respectively.

Revised: In the end of each stage, there will be a down-sampling operation to reduce the resolution of the feature map by 2.

Revised: We collect the feature map before the down-sampling operation as the output of the stage.

Revised: In the end, the strides of output feature maps are 4, 8, 16, and 32 respectively.

Revised: And the channels are 192, 384, 768, and 1536, respectively.

Revised: (b) Up-Sampling 32->

Revised: Given the feature maps

Revised: of strides

Revised: and

Revised: from (a) , we

Revised: first

Revised: up-

Revised: sample the feature map

Revised: of stride

Revised: to stride

Revised: 16 via

Revised: bi-linear interpolation

Revised: Then, we

Revised: concatenate

Revised: the feature maps and obtain

Revised: a new

Revised: feature

Revised: map of stride

Revised: and

Revised: channels 1536 +

Revised: =

Revised: We apply

Revised: a convolution layer that

Revised: has

Revised: 2304 input channels, 2304 output channels, and 4 groups,

Revised: to

Revised: fuse

Revised: the in

Revised: - formation.

Revised: We

Revised: also append an instance normalization layer (Ulyanov et al., 2016) and a LeakyReLU activation function.

Revised: Next, we apply another convolution layer

Revised: with

Revised: as 2304 input channels, 512 out put channels to compress the feature channels.

Revised: Again we append an instance normalization layer and the LeakyReLU activation function.

Revised: In the end, we add a skip connection between the result feature map and the previous concatenated feature map by addition operation.

Revised: The concatenated feature map will be transformed to consistent number of channels in advance by a convolution layer with 2304 input channels and 512 output channels.

Revised: The final feature map has stride 16 andchannels.

Revised: (c) V-Layer.

Revised: The V-layer takes the output feature map from (b) as input.

Revised: We first utilize a convolution layer with 512 input channels and 512 output channels to transform the feature map into a more appropriate hidden space.

Revised: The feature is transformed by the LeakyReLU activation function.

Revised: Then we utilize another convolution layer with 512 input channels as and 2 × 16 = 32 output channels to predict the gradients along the horizontal and vertical axis, respectively.

Revised: The number 16 represents that we predict 16 channels of horizontal gradients and 16 channels of vertical gradients respectively, where each channel is expected to capture different details in scenes.

Revised: Similarly, we also use another convolution layer with 512 input channels, 2 × 16 = 32 output channels to predict the corresponding weight.

Revised: The weight will be transformed by the Sigmoid function.

Revised: Then we reshape the predictions and compute the unscaled depth map for each pair of gradient and weight following Eq.(5).

Revised: The unscaled depth maps computed from all the pairs will be concatenated along the channel dimension.

Revised: Thereby we obtain a depth map with 16 channels and stride 16.

Revised: We apply a group normalization on the depth map, where the 16 channels are viewed as a single group.

Revised: In the end, we expand the channels of the depth map into 128 by a convolution layer with 16 input channels and 128 output channels.

Revised: Thereby the output of the V-layer is a depth map with 128 channels and stride 16.

Revised: (d) Refine 16.

Revised: We refine the feature map from (b) and the depth map from (c) .

Revised: Specifically, we first concatenate the feature map with depth map, and obtain a new feature map with 512 + 128 =channels.

Revised: Then we employ a convolution layer with 640 input channels and 640 output channels to fuse the information.

Revised: The result feature

Revised: map will be further transformed by the LeakyReLU activation function.

Revised: Next a convolution layer with 640 input channels and 128 output channels is applied to predict the refined depth map.

Revised: Similarly, we also predict the refined feature map by a convolution layer with 640 input channels and 512 output channels.

Revised: (e) Up-Sampling 16->8.

Revised: Here, we up-sample the feature map from (d) to stride 8 via bi-linear interpolation.

Revised: To recover the details of scenes, we concatenate with the feature map from the encoder that has the same stride.

Revised: In such a way, we obtain a new feature map with 384+512 = 896 channels and stride 8.

Revised: Similar to (b) , we apply a convolution layer with 896 input channels, 896 output channels, and 4 groups to fuse the information.

Revised: Then the feature map is transformed by the instance normalization layer and the LeakyReLU activation function.

Revised: Next, we apply another convolution

Revised: layer with 896 input channels and 256 output channels to compress feature channels.

Revised: Again, we apply the instance normalization layer and LeakyReLU activation function.

Revised: Same as (b) , we add a skip connection between the result feature map and the previous concatenated feature map by addition operation.

Revised: The concatenated feature map will be transformed to the consistent number of channels by a convolution layer with 896 input channels and 256 output channels.

Revised: Thereby the final feature map has stride 8 and 256 channels.

Revised: (f) Refine 8.

Revised: We refine the depth map from (d) and the feature map from (e) .

Revised: The procedure is the same as (d) , except for the number of channels in convolution layers are adapted accordingly.

Revised: Thereby we only briefly introduce the pipeline.

Revised: We first up-sample the depth map from (d) into stride 8 to concatenate with the feature map.

Revised: Thereby we obtain a new feature map with 256 + 128 =channels.

Revised: Then, we apply a convolution layer with 384 input channels and 384 output channels to fuse the information.

Revised: The feature map is then transformed by LeakyReLU activation function.

Revised: Next, we utilize a convolution layer with 384 input channels and 128 output channels to predict the refined depth map.

Revised: Similarly, another convolution layer with 384 input channels and 256 output channels is applied to predict the refined feature map.

Revised: (g) Up-Sampling 8->4.

Revised: We up-sample the feature map from (f) to stride 4.

Revised: The procedure is the same as (e) except for the change of the number of channels.

Revised: More specifically, we first upsample the feature map and concatenate with the feature map from the encoder with consistent stride, obtaining a feature map with 256 + 192 = 448 channels and stride 4.

Revised: Then a convolution layer with 448 input channels, 448 output channels, and 4 groups is applied to fuse the information.

Revised: Then the result feature map is transformed by the instance normalization and LeakyReLU activation function.

Revised: Next, we apply another convolution layer with 448 input channels and 64 output channels to compress the feature channels.

Revised: Again, the instance normalization layer and LeakyReLU function is applied.

Revised: In the end, we add a skip connection between the final feature map and the previous concatenated feature map.

Revised: (h) Refine 4.

Revised: We refine the depth map from (f) and the feature map from (g) following the same pipeline as (d) .

Revised: We fist up-sample the depth map to stride 4, and concatenate with the feature map, obtaining a new feature map with 64 + 128 = 192 channels and stride 4.

Revised: Next, we apply a convolution layer with 192 input channels and 192 output channels to fuse the information, and a

Revised: LeakyReLU function to transform the feature value.

Revised: In the end, we utilize a convolution layer with 192 input channels and 128 output channel to predict the new depth map.

Revised: The output is a depth map with 128 channels and stride 4.

Revised: (i) Metric Layer.

Revised: We take the feature map with stride 32 from the encoder as input.

Revised: We first apply global max pooling to compress the feature map into a vector with 1536 channels.

Revised: Then we apply a fully-connected layer with 1536 input units and 384 output units to compress the channels.

Revised: Next we apply a LeakyReLU function.

Revised: In the end, we utilize another fully-connected layer with 384 input units and 2 output units to regress the scale and shift.

Revised: (j) Final Prediction.

Revised: We take the output from (d) , (f) , (h) and (i) as input.

Revised: We up-sample all the depth maps to stride 1 via bi-linear interpolation, and fuse the depth maps into a single depth map by a convolution layer with 128 input channels and 1 output channel.

Revised: In the end, we add the depth map with the shift and multiply with the scale from (i) , respectively.

Revised: A. S TATISTICAL E VALUATION W ITH O THER M ETHODS

Revised: In this section, we compare with Ramamonjisoa et al.

Revised: (2020), Cheng et al.

Revised: (2018), and Li et al.

Revised: (2017) on NYU Depth V2 (Silberman et al., 2012).

Revised: More specifically, we apply their methods to refine the final predicted depth map.

Revised: The displacement field, the affinity matrix, and the depth gradient are predicted from the refined feature map, which is the output of the last refinement module.

Revised: For

Revised: Cheng et al.

Revised: we adopt their default hyper-parameters.

Revised: The kernel size is 3, and the number of iterations is 24.

Revised: Li et al.

Revised: (2017) proposed the end-to-end fusion strategy and the optimization-based fusion strategy.

Revised: For the end-to-end fusion strategy, we employ three convolution layers with kernel size 5, and 16 channels.

Revised: For the optimization-based fusion strategy, we use Adam optimizer.

Revised: The learning rate is 1e-2 and the number of iterations is 100.

Revised: The results are shown in Tab.

Revised: Our method achieves better performance.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: A. A BLATION ON R ESOLUTION

Revised: In this part, we evaluate the performance of the V-layer when operating on the feature map with stride 8.

Revised: We achieve this by dividing the feature map into 4 rectangle sub-regions with equal area (top left, top right, bottom left and bottom right).

Revised: We apply a V-layer on each sub-region.

Revised: The predicted depth map from the sub-regions will be stacked to recover the original resolution.

Revised: The results are shown in Tab. 10.

Revised: From stride 16 to 8, the improvement of accuracy is marginal.

Revised: However, the inference speed (FPS) is significantly slow down.

Original: To further demonstrate the benefits of our V-layer on KITTI Eigen split (Geiger et al., 2012; Eigen et al., 2014) and SUN RGB-D test set (Song et al., 2015), we present the performance of the predicted depth map when replacing the V-layer with a single convolutional layer.
Revised: To further demonstrate the benefits of our V-layer on KITTI Eigen split (Geiger et al., 2012; Eigen et al., 2014) and SUN RGB-D test set (Song et al., 2015) [Table] we present the performance of the predicted depth map when replacing the V-layer with a single convolutional layer.

Original: As shown in Tab.(9), our V-layer can improve the depth map accuracy on both KITTI and SUN
Revised: As shown in Tab.(11) [Table] our V-layer can improve the depth map accuracy on both KITTI and SUN

Original: RGB-D.
Revised: RGB-D.

Original: [Table]
Revised: [Table]

Revised: A. M ORE G ENERALIZATION E XPERIMENTS

Original: [Figure]
Revised: We further evaluated the generalization performance when training on KITTI and test on NYU, or training on NYU and test on KITTI. The results are shown in Tab. 12 and Tab. 13. Our method achieves better generalization performance in both settings.

Original: A. M ORE V ISUALIZATION
Revised: A. M ORE V ISUALIZATION

Original: We visualize the confidence weight Σ x , the difference map Γ x , and the depth map Z u from the
Revised: We visualize the confidence weight Σ x , the difference map Γ x , and the depth map Z u from the

Original: V-layer in Fig.6.
Revised: V-layer in Fig.5.

Original: We observe that the depth value of a pixel shows correlation with respect to the image coordinates of the pixel.
Revised: We observe that the depth value of a pixel shows correlation with respect to the image coordinates of the pixel.

Original: For example, in the last example in Fig.6, for different pixels at the door, the depth values are usually different but the first order difference are approximately the same.
Revised: For example, in the last example in Fig.5, for different pixels at the door, the depth values are usually different but the first order difference are approximately the same.

Original: This observation shows that the difference map might be easier to predict than the depth map.
Revised: This observation shows that the difference map might be easier to predict than the depth map.

Original: A. Q UALITATIVE R ESULTS
Revised: A. Q UALITATIVE R ESULTS

Original: We provide more qualitative results on KITTI Eigen split (Eigen et al., 2014), SUN RGB-D (Song et al., 2015), and NYU Depth V2 (Silberman et al., 2012) in Fig.7, Fig.8 and Fig.9, respectively.
Revised: We provide more qualitative results on KITTI Eigen split (Eigen et al., 2014), SUN RGB-D (Song et al., 2015), and NYU Depth V2 (Silberman et al., 2012) in Fig.6, Fig.7 and Fig.8, respectively.

Original: Our framework can predict more accurate shapes and preserve the high-frequency scene details.
Revised: Our framework can predict more accurate shapes and preserve the high-frequency scene details.

Original: [Figure] [Figure]
Revised: [Table] [Table] [Figure]

Original: [Figure] 
Revised: [Figure] [Figure] [Figure]