Original: VA-D EPTH N ET : A V ARIATIONAL A PPROACH TO S INGLE I MAGE D EPTH P REDICTION
Revised: VA-D EPTH N ET : A V ARIATIONAL A PPROACH TO S INGLE I MAGE D EPTH P REDICTION

Original: A BSTRACT
Revised: A BSTRACT

Original: We introduce VA-DepthNet, a simple, effective, and accurate deep neural network approach for the single-image depth prediction (SIDP) problem.
Revised: We introduce VA-DepthNet, a simple, effective, and accurate deep neural network approach for the single-image depth prediction (SIDP) problem.

Original: The proposed approach advocates using classical first-order variational constraints for this problem.
Revised: The proposed approach advocates using classical first-order variational constraints for this problem.

Original: While state-of-the-art deep neural network methods for SIDP learn the scene depth from images in a supervised setting, they often overlook the invaluable invariances and priors in the rigid scene space, such as the regularity of the scene.
Revised: While state-of-the-art deep neural network methods for SIDP learn the scene depth from images in a supervised setting, they often overlook the invaluable invariances and priors in the rigid scene space, such as the regularity of the scene.

Original: The paper’s main contribution is to reveal the benefit of classical and well-founded variational constraints in the neural network design for the SIDP task.
Revised: The paper’s main contribution is to reveal the benefit of classical and well-founded variational constraints in the neural network design for the SIDP task.

Original: It is shown that imposing first-order variational constraints in the scene space together with popular encoder-decoder-based network architecture design provides excellent results for the supervised SIDP task.
Revised: It is shown that imposing first-order variational constraints in the scene space together with popular encoder-decoder-based network architecture design provides excellent results for the supervised SIDP task.

Original: The imposed first-order variational constraint makes the network aware of the depth gradient in the scene space, i.e., regularity.
Revised: The imposed first-order variational constraint makes the network aware of the depth gradient in the scene space, i.e., regularity.

Original: The paper demonstrates the usefulness of the proposed approach via extensive evaluation and ablation analysis over several benchmark datasets, such as KITTI,
Revised: The paper demonstrates the usefulness of the proposed approach via extensive evaluation and ablation analysis over several benchmark datasets, such as KITTI,

Original: NYU Depth V2, and SUN RGB-D. The VA-DepthNet at test time shows considerable improvements in depth prediction accuracy compared to the prior art and is accurate also at high-frequency regions in the scene space.
Revised: NYU Depth V2, and SUN RGB-D. The VA-DepthNet at test time shows considerable improvements in depth prediction accuracy compared to the prior art and is accurate also at high-frequency regions in the scene space.

Original: At the time of writing this paper, our method—labeled as VA-DepthNet, when tested on the KITTI depth-prediction evaluation set benchmarks, shows state-of-the-art results, and is
Revised: At the time of writing this paper, our method—labeled as VA-DepthNet, when tested on the KITTI depth-prediction evaluation set benchmarks, shows state-of-the-art results, and is

Original: the
Revised: the

Original: top performing published approach 1 2 .
Revised: top performing published approach 1 2 .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Over the last decade, neural networks have introduced a new prospect for the 3D computer vision field.
Revised: Over the last decade, neural networks have introduced a new prospect for the 3D computer vision field.

Original: It has led to significant progress on many long-standing problems in this field, such as multiview stereo (Huang et al., 2018; Kaya et al., 2022), visual simultaneous localization and mapping (Teed & Deng, 2021), novel view synthesis (Mildenhall et al., 2021), etc.
Revised: It has led to significant progress on many long-standing problems in this field, such as multiview stereo (Huang et al., 2018; Kaya et al., 2022), visual simultaneous localization and mapping (Teed & Deng, 2021), novel view synthesis (Mildenhall et al., 2021), etc.

Original: Among several 3D vision problems, one of the challenging, if not impossible, to solve is the single-image depth prediction (SIDP) problem.
Revised: Among several 3D vision problems, one of the challenging, if not impossible, to solve is the single-image depth prediction (SIDP) problem.

Original: SIDP is indeed ill-posed—in a strict geometric sense, presenting an extraordinary challenge to solve this inverse problem reliably.
Revised: SIDP is indeed ill-posed—in a strict geometric sense, presenting an extraordinary challenge to solve this inverse problem reliably.

Original: Moreover, since we do not have access to multi-view images, it is hard to constrain this problem via well-known geometric constraints (Longuet-Higgins, 1981; Nistér, 2004; Furukawa & Ponce, 2009; Kumar et al., 2019; 2017).
Revised: Moreover, since we do not have access to multi-view images, it is hard to constrain this problem via well-known geometric constraints (Longuet-Higgins, 1981; Nistér, 2004; Furukawa & Ponce, 2009; Kumar et al., 2019; 2017).

Original: Accordingly, the SIDP problem generally boils down to an ambitious fitting problem, to which deep learning provides a suitable way to predict an acceptable solution to this problem (Yuan et al., 2022; Yin et al., 2019).
Revised: Accordingly, the SIDP problem generally boils down to an ambitious fitting problem, to which deep learning provides a suitable way to predict an acceptable solution to this problem (Yuan et al., 2022; Yin et al., 2019).

Original: Impressive earlier methods use Markov Random Fields (MRF) to model monocular cues and the relation between several over-segmented image parts (Saxena et al., 2007; 2008).
Revised: Impressive earlier methods use Markov Random Fields (MRF) to model monocular cues and the relation between several over-segmented image parts (Saxena et al., 2007; 2008).

Original: Nevertheless, with the recent surge in neural network architectures (Krizhevsky et al., 2012; Simonyan & Zisserman, 2015; He et al., 2016), which has an extraordinary capability to perform complex regression, many current works use deep learning to solve SIDP and have demonstrated high-quality results (Yuan et al., 2022;
Revised: Nevertheless, with the recent surge in neural network architectures (Krizhevsky et al., 2012; Simonyan & Zisserman, 2015; He et al., 2016), which has an extraordinary capability to perform complex regression, many current works use deep learning to solve SIDP and have demonstrated high-quality results (Yuan et al., 2022;

Original: Aich et al., 2021;
Revised: Aich et al., 2021;

Original: Bhat et al., 2021; Eigen et al., 2014; Fu et al., 2018; Lee et al., 2019;
Revised: Bhat et al., 2021; Eigen et al., 2014; Fu et al., 2018; Lee et al., 2019;

Original: Popular recent methods for SIDP are mostly supervised.
Revised: Popular recent methods for SIDP are mostly supervised.

Original: But even then, they are used less in real-world applications than geometric multiple view methods (Labbé & Michaud, 2019; Müller et al., 2022).
Revised: But even then, they are used less in real-world applications than geometric multiple view methods (Labbé & Michaud, 2019; Müller et al., 2022).

Original: Nonetheless, a good solution to SIDP is highly desirable in robotics (Yang et al., 2020), virtual-reality (Hoiem et al., 2005), augmented reality (Du et al., 2020), view synthesis (Hoiem et al., 2005) and other related vision tasks (Liu et al., 2019).
Revised: Nonetheless, a good solution to SIDP is highly desirable in robotics (Yang et al., 2020), virtual-reality (Hoiem et al., 2005), augmented reality (Du et al., 2020), view synthesis (Hoiem et al., 2005) and other related vision tasks (Liu et al., 2019).

Original: In this paper, we advocate that despite the supervised approach being encouraging, SIDP advancement should not wholly rely on the increase of dataset sizes.
Revised: In this paper, we advocate that despite the supervised approach being encouraging, SIDP advancement should not wholly rely on the increase of dataset sizes.

Original: Instead, geometric cues and scene priors could help improve the SIDP results.
Revised: Instead, geometric cues and scene priors could help improve the SIDP results.

Original: Not that scene priors have not been studied to improve
Revised: Not that scene priors have not been studied to improve

Original: SIDP accuracy in the past.
Revised: SIDP accuracy in the past.

Original: For instance, Chen et al.
Revised: For instance, Chen et al.

Original: (2016) uses pairwise ordinal relations between points to learn scene depth.
Revised: (2016) uses pairwise ordinal relations between points to learn scene depth.

Original: Alternatively, Yin et al.
Revised: Alternatively, Yin et al.

Original: (2019) uses surface normals as an auxiliary loss to improve performance.
Revised: (2019) uses surface normals as an auxiliary loss to improve performance.

Original: Other heuristic approaches, such as Qi et al.
Revised: Other heuristic approaches, such as Qi et al.

Original: (2018), jointly exploit the depth-to-normal relation to recover scene depth and surface normals.
Revised: (2018), jointly exploit the depth-to-normal relation to recover scene depth and surface normals.

Original: Yet, such state-of-the-art SIDP methods have limitations: for example, the approach in Chen et al.
Revised: Yet, such state-of-the-art SIDP methods have limitations: for example, the approach in Chen et al.

Original: (2016) - using ordinal relation to learn depth - over-smooths the depth prediction results, thereby failing to preserve high-frequency surface details.
Revised: (2016) - using ordinal relation to learn depth - over-smooths the depth prediction results, thereby failing to preserve high-frequency surface details.

Original: Conversely, Yin et al.
Revised: Conversely, Yin et al.

Original: (2019) relies on good depth map prediction from a deep network and the idea of virtual normal.
Revised: (2019) relies on good depth map prediction from a deep network and the idea of virtual normal.

Original: The latter is computed by randomly sampling three noncollinear points with large distances.
Revised: The latter is computed by randomly sampling three noncollinear points with large distances.

Original: This is rather complex and heuristic in nature.
Revised: This is rather complex and heuristic in nature.

Original: Qi et al.
Revised: Qi et al.

Original: (2018) uses depth and normal consistency, which is good, yet it requires good depth map initialization.
Revised: (2018) uses depth and normal consistency, which is good, yet it requires good depth map initialization.

Original: This brings us to the point that further generalization of the regression-based SIDP pipeline is required.
Revised: This brings us to the point that further generalization of the regression-based SIDP pipeline is required.

Original: As mentioned before, existing approaches in this direction have limitations and are complex.
Revised: As mentioned before, existing approaches in this direction have limitations and are complex.

Original: In this paper, we propose a simple approach that provides better depth accuracy and generalizes well across different scenes.
Revised: In this paper, we propose a simple approach that provides better depth accuracy and generalizes well across different scenes.

Original: To this end, we resort to the physics of variation (Mollenhoff et al., 2016;
Revised: To this end, we resort to the physics of variation (Mollenhoff et al., 2016;

Original: Chambolle et al., 2010) in the neural network design for better generalization of the SIDP network, which by the way, keeps the essence of affine invariance (Yin et al., 2019).
Revised: Chambolle et al., 2010) in the neural network design for better generalization of the SIDP network, which by the way, keeps the essence of affine invariance (Yin et al., 2019).

Original: An image of a general scene—indoor or outdoor, has a lot of spatial regularity.
Revised: An image of a general scene—indoor or outdoor, has a lot of spatial regularity.

Original: And therefore, introducing a variational constraint provides a convenient way to ensure spatial regularity and to preserve information related to the scene discontinuities (Chambolle et al., 2010).
Revised: And therefore, introducing a variational constraint provides a convenient way to ensure spatial regularity and to preserve information related to the scene discontinuities (Chambolle et al., 2010).

Original: Consequently, the proposed network is trained in a fully-supervised manner while encouraging the network to be mindful of the scene regularity where the variation in the depthgradient is large (cf. Sec.3.1).
Revised: Consequently, the proposed network is trained in a fully-supervised manner while encouraging the network to be mindful of the scene regularity where the variation in the depth is large (cf. Sec.3.1).

Original: In simple terms, depth regression must be more than parameter fitting, and at some point, a mindful decision must be made—either by imaging features or by scene depthgradient variation, or both.
Revised: In simple terms, depth regression must be more than parameter fitting, and at some point, a mindful decision must be made—either by imaging features or by scene depth variation, or both.

Original: As we demonstrate later in the paper, such an idea boosts the network’s depth accuracy while preserving the high-frequency and low-frequency scene information (see Fig.1).
Revised: As we demonstrate later in the paper, such an idea boosts the network’s depth accuracy while preserving the high-frequency and low-frequency scene information (see Fig.1).

Original: Our neural network for SIDP disentangles the absolute scale from the metric depth.
Revised: Our neural network for SIDP disentangles the absolute scale from the metric depth map.

Original: It models an unscaled depth map as the optimal solution to the pixel-level variational constraints via weighted firstorder differences, respecting the neighboring pixel depth gradients.
Revised: It models an unscaled depth map as the optimal solution to the pixel-level variational constraints via weighted first-order differences, respecting the neighboring pixel depth gradients.

Original: Compared to previous methods, the network’s task has been shifted away from pixel-wise metric depth learning to learning the firstorder differences of the scene, which alleviates the scale ambiguity and favors scene regularity.
Revised: Compared to previous methods, the network’s task has been shifted away from pixel-wise metric depth learning to learning the first-order differences of the scene, which alleviates the scale ambiguity and favors scene regularity.

Original: To realize that, we initially employ a neural network to predict the first-order differences of the depth map.
Revised: To realize that, we initially employ a neural network to predict the first-order differences of the depth map.

Original: Then, we construct the partial differential equations representing the variational constraints by reorganizing the differences into a large matrix, i.e., an over-determined system of equations.
Revised: Then, we construct the partial differential equations representing the variational constraints by reorganizing the differences into a large matrix, i.e., an over-determined system of equations.

Original: Further, the network learns a weight matrix to eliminate redundant equations that do not favor the introduced first-order difference constraint.
Revised: Further, the network learns a weight matrix to eliminate redundant equations that do not favor the introduced first-order difference constraint.

Original: Finally, the closed-form depth map solution is recovered via simple matrix operations.
Revised: Finally, the closed-form depth map solution is recovered via simple matrix operations.

Original: When tested on the KITTI (Geiger et al., 2012) and NYU Depth V2 (Silberman et al., 2012) test sets, our method outperforms prior art depth prediction accuracy by a large margin.
Revised: When tested on the KITTI (Geiger et al., 2012) and NYU Depth V2 (Silberman et al., 2012) test sets, our method outperforms prior art depth prediction accuracy by a large margin.

Original: Moreover, our model pre-trained on NYU Depth V2 better generalizes to the SUN RGB-D test set.
Revised: Moreover, our model pre-trained on NYU Depth V2 better generalizes to the SUN RGB-D test set.

Original:  P RIOR W ORK
Revised:  P RIOR W ORK

Original: Depth estimation is a longstanding task in computer vision.
Revised: Depth estimation is a longstanding task in computer vision.

Original: In this work, we focus on a fullysupervised, single-image approach, and therefore, we discuss prior art that directly relates to such approach.
Revised: In this work, we focus on a fullysupervised, single-image approach, and therefore, we discuss prior art that directly relates to such approach.

Original: Broadly, we divide the popular supervised SIDP methods into three sub-categories.
Revised: Broadly, we divide the popular supervised SIDP methods into three sub-categories.

Original: (i) Depth Learning using Ranking or Ordinal Relation Constraint.
Revised: i Depth Learning using Ranking or Ordinal Relation Constraint.

Original: Zoran et al.
Revised: Zoran et al.

Original: and
Revised: and

Original: Chen et al.
Revised: Chen et al.

Original: (2016) argue that the ordinal relation between points is easier to learn than the metric depth.
Revised: (2016) argue that the ordinal relation between points is easier to learn than the metric depth.

Original: To this end, Zoran et al.
Revised: To this end , Zoran et al

Original: (2015) proposes constrained quadratic optimization while Chen
Revised: (2015) proposes constrained quadratic optimization while Chen

Original: [Figure]
Revised: [Figure]

Original: et al.
Revised: et al.

Original: (2016) relies on the variation of the inception module to solve the problem.
Revised: (2016) relies on the variation of the inception module to solve the problem.

Original: Later, Xian et al.
Revised: Later, Xian et al.

Original: (2020) proposes structure-guided sampling strategies for point pairs to improve training efficiency.
Revised: (2020) proposes structure-guided sampling strategies for point pairs to improve training efficiency.

Original: Recently, Lienen et al.
Revised: Recently, Lienen et al.

Original: (2021) elaborates on the use of listwise ranking method based on the PlackettLuce model (Luce, 2012).
Revised: (2021) elaborates on the use of listwise ranking method based on the PlackettLuce model (Luce, 2012).

Original: The drawback of such approaches is that the ordinal relationship and ranking over smooth the depth solution making accurate metric depth recovery challenging.
Revised: The drawback of such approaches is that the ordinal relationship and ranking over smooth the depth solution making accurate metric depth recovery challenging.

Original: (ii) Depth Learning using Surface Normal Constraint.
Revised: (ii) Depth Learning using Surface Normal Constraint.

Original: Hu et al.
Revised: Hu et al.

Original: (2019) introduces normal loss in addition to the depth loss to overcome the distorted and blurry edges in the depth prediction.
Revised: (2019) introduces normal loss in addition to the depth loss to overcome the distorted and blurry edges in the depth prediction.

Original: Yin et al.
Revised: Yin et al.

Original: (2019) proposes the concept of virtual normal to impose 3D scene constraint explicitly and to capture the long-range relations in the depth prediction.
Revised: (2019) proposes the concept of virtual normal to impose 3D scene constraint explicitly and to capture the long-range relations in the depth prediction.

Original: The long-range dependency in 3D is introduced via random sampling of three non-colinear points at a large distance from the virtual plane.
Revised: The long-range dependency in 3D is introduced via random sampling of three non-colinear points at a large distance from the virtual plane.

Original: Lately, Long et al.
Revised: Lately, Long et al.

Original: (2021) proposes an adaptive strategy to compute the local patch surface normals at train time from a set of randomly sampled candidates and overlooks it during test time.
Revised: (2021) proposes an adaptive strategy to compute the local patch surface normals at train time from a set of randomly sampled candidates and overlooks it during test time.

Original: (iii) Depth Learning using other Heuristic Refinement Constraint.
Revised: (iii) Depth Learning using other Heuristic Refinement Constraint.

Original: There has been numerous works attempt to refine the depth prediction as a post-processing step.
Revised: There has been numerous works attempt to refine the depth prediction as a post-processing step.

Original: Liu et al. (2015), Li et al.
Revised: Liu et al. (2015), Li et al.

Original: (2015) and Yuan et al.
Revised: (2015) and Yuan et al.

Original: (2022) propose to utilize the Conditional Random Fields (CRF) to smooth the depth map.
Revised: (2022) propose to utilize the Conditional Random Fields (CRF) to smooth the depth map.

Original: Lee et al.
Revised: Lee et al.

Original: (2019) utilizes the planar assumptions to regularize the predicted depth map.
Revised: (2019) utilizes the planar assumptions to regularize the predicted depth map.

Original: Qi et al.
Revised: Qi et al.

Original: (2018) adopts an auxiliary network to predict the surface normal, and then refine the predicted depth map following their proposed heuristic rules.
Revised: (2018) adopts an auxiliary network to predict the surface normal, and then refine the predicted depth map following their proposed heuristic rules.

Original: There are mainly two problems with such approaches: Firstly, these approaches rely on a good depth map initialization.
Revised: There are mainly two problems with such approaches: Firstly, these approaches rely on a good depth map initialization.

Original: Secondly, the heuristic rules and the assumptions might result in over-smoothed depth values at objects boundaries.
Revised: Secondly, the heuristic rules and the assumptions might result in over-smoothed depth values at objects boundaries.

Original: Meanwhile, a few works, such as Ramamonjisoa et al.
Revised: Meanwhile, a few works, such as Ramamonjisoa et al.

Original: (2020), Cheng et al.
Revised: (2020), Cheng et al.

Original: , Li et al.
Revised: , Li et al.

Original: (2017) were proposed in the past with similar inspirations.
Revised: (2017) were proposed in the past with similar inspirations.

Original: Ramamonjisoa et al.
Revised: Ramamonjisoa et al.

Original: (2020), Cheng et al.
Revised: (2020), Cheng et al.

Original: (2018) methods are generally motivated towards depth map refinement predicted from an off-the-shelf network.
Revised: (2018) methods are generally motivated towards depth map refinement predicted from an off-the-shelf network.

Original: On the other hand, Cheng et al.
Revised: On the other hand, Cheng et al.

Original: (2018) proposes to use an affinity matrix that aims to learn the relation between each pixel’s depth value and its neighbors’ depth values.
Revised: (2018) proposes to use an affinity matrix that aims to learn the relation between each pixel’s depth value and its neighbors’ depth values.

Original: However, the affinity matrix has no explicit supervision, which could lead to imprecise learning of neighboring relations providing inferior results.
Revised: However, the affinity matrix has no explicit supervision, which could lead to imprecise learning of neighboring relations providing inferior results.

Original: On the contrary, our approach is mindful of imposing the first-order difference constraint leading to better performance.
Revised: On the contrary, our approach is mindful of imposing the first-order difference constraint leading to better performance.

Original: Earlier, Li et al.
Revised: Earlier, Li et al.

Original: (2017) proposed two strategies for SIDP, i.e., fusion in an end-to-end network and fusion via optimization.
Revised: (2017) proposed two strategies for SIDP, i.e., fusion in an end-to-end network and fusion via optimization.

Original: The end-to-end strategy fuses the gradient and the depth map via convolution layers without any constraint on convolution weights, which may not be an apt choice for a depth regression problem such as SIDP.
Revised: The end-to-end strategy fuses the gradient and the depth map via convolution layers without any constraint on convolution weights, which may not be an apt choice for a depth regression problem such as SIDP.

Original: On the other hand, the fusion via optimization strategy is based on a non-differentiable strategy, leading to a nonend-to-end network loss function.
Revised: On the other hand, the fusion via optimization strategy is based on a non-differentiable strategy, leading to a nonend-to-end network loss function.

Original: Contrary to that, our method is well-constrained and performs quite well with a loss function that helps end-to-end learning of our proposed network.
Revised: Contrary to that, our method is well-constrained and performs quite well with a loss function that helps end-to-end learning of our proposed network.

Original: Not long ago, Lee & Kim (2019) proposed to estimate relative depths between pairs of images and ordinary depths at a different scale.
Revised: Not long ago, Lee & Kim (2019) proposed to estimate relative depths between pairs of images and ordinary depths at a different scale.

Original: By exploiting the rank-1 property of the pairwise comparison matrix, it recovers the relative depth map.
Revised: By exploiting the rank-1 property of the pairwise comparison matrix, it recovers the relative depth map.

Original: Later, relative and ordinary depths are decomposed and fused to recover the depth.
Revised: Later, relative and ordinary depths are decomposed and fused to recover the depth.

Original: On a slightly different note, Lee & Kim (2020) studies the effectiveness of various losses and how to combine them for better monocular depth prediction.
Revised: On a slightly different note, Lee & Kim (2020) studies the effectiveness of various losses and how to combine them for better monocular depth prediction.

Original: To sum up, our approach allows learning of confidence weight to select reliable gradient estimation in a fully differentiable manner.
Revised: To sum up, our approach allows learning of confidence weight to select reliable gradient estimation in a fully differentiable manner.

Original: Further, it proffers the benefits of the variational approach to overcome the limitations of the existing state-of-the-art methods.
Revised: Further, it proffers the benefits of the variational approach to overcome the limitations of the existing state-of-the-art methods.

Original: More importantly, the proposed method can provide excellent depth prediction without making extra assumptions such as good depth initialization, piece-wise planar scene, and assumptions used by previous works mentioned above.
Revised: More importantly, the proposed method can provide excellent depth prediction without making extra assumptions such as good depth initialization, piece-wise planar scene, and assumptions used by previous works mentioned above.

Original: [Figure]
Revised: [Figure]

Original:  M ETHOD
Revised:  M ETHOD

Original: In this section, we first describe our proposed variational constraint and then present the overall network architecture leading to the overall loss function.
Revised: In this section, we first describe our proposed variational constraint and then present the overall network architecture leading to the overall loss function.

Original: 3. V ARIATIONAL C ONSTRAINT
Revised: 3. V ARIATIONAL C ONSTRAINT

Original: Here we introduce our variational constraint and how it can be useful for depth estimation.
Revised: Here we introduce our variational constraint and how it can be useful for depth estimation.

Original: Consider an unscaled depth map as Z u ∈ R H × W , with ( H, W ) symbolizing the height and width, respectively.
Revised: Consider an unscaled depth map as Z u ∈ R H × W , with ( H, W ) symbolizing the height and width, respectively.

Original: Assuming Γ x ∈ R H × W and Γ y ∈ R H × W as the gradient of Z u in the x and y axis, we write
Revised: Assuming Γ x ∈ R H × W and Γ y ∈ R H × W as the gradient of Z u in the x and y axis, we write

Original: [Equation]
Revised: [Equation]

Original: Here, x and y subscript corresponds to the direction from left to right ( x -axis) and top to bottom of the image ( y -axis), respectively.
Revised: Here, x and y subscript corresponds to the direction from left to right ( x -axis) and top to bottom of the image ( y -axis), respectively.

Original: Elaborating on this, we can write
Revised: Elaborating on this, we can write

Original: [Equation]
Revised: [Equation]

Original: Suppose we augment Eq.(2) expression for all ( i, j ) , i ∈ { 1 , ..., H } and j ∈ { 1 , ..., W } .
Revised: Suppose we augment Eq.(2) expression for all ( i, j ) , i ∈ { 1 , ..., H } and j ∈ { 1 , ..., W } .

Original: In that case, we will end up with an over-determined system with 2 HW equations in total.
Revised: In that case, we will end up with an over-determined system with 2 HW equations in total.

Original: Given the predicted Γ x and Γ y , we aim to recover the HW unknown variables in Z u .
Revised: Given the predicted Γ x and Γ y , we aim to recover the HW unknown variables in Z u .

Original: However, some of the equations could be spurious and deteriorate the overall depth estimation result rather than improving it.
Revised: However, some of the equations could be spurious and deteriorate the overall depth estimation result rather than improving it.

Original: As a result, we must be mindful about selecting the equation that respects the imposed first-order constraint and maintains the depth gradient to have a meaningful fitting for better generalization.
Revised: As a result, we must be mindful about selecting the equation that respects the imposed first-order constraint and maintains the depth gradient to have a meaningful fitting for better generalization.

Original: To that end, we introduce confidence weight Σ x ∈ [0 , 1] H × W , Σ y ∈ [0 , 1] H × W for gradient along x, y direction.
Revised: To that end, we introduce confidence weight Σ x ∈ [0 , 1] H × W , Σ y ∈ [0 , 1] H × W for gradient along x, y direction.

Original: Consequently, we multiply the above two equations by the confidence weight term Σ
Revised: Consequently, we multiply the above two equations by the confidence weight term Σ

Original: i,jx and Σ i,jy , respectively.
Revised: i,jx and Σ i,jy , respectively.

Original: On one hand, if the confidence is close to 1, the equation will have priority to be satisfied by the optimal Z u .
Revised: On one hand, if the confidence is close to 1, the equation will have priority to be satisfied by the optimal Z u .

Original: On the other hand, if the confidence is close to 0, we must ignore the equation.
Revised: On the other hand, if the confidence is close to 0, we must ignore the equation.

Original: For better understanding, we illustrate the first-order difference and weighted matrix construction in Fig.2 (a) and Fig.2 (b).
Revised: For better understanding, we illustrate the first-order difference and weighted matrix construction in Fig.2 (a) and Fig.2 (b).

Original: Next, we reshape the Σ x , Σ y , Γ x , Γ y , and Z u into column vectors ˜Σ x ∈ [0 , 1] HW × 1 , ˜Σ y ∈ [0 , 1] HW × 1 , ˜Γ x ∈ R HW × 1 , ˜Γ y ∈ R HW × 1 , and ˜ Z u ∈ R HW × 1 , respectively.
Revised: Next, we reshape the Σ x , Σ y , Γ x , Γ y , and Z u into column vectors ˜Σ x ∈ [0 , 1] HW × 1 , ˜Σ y ∈ [0 , 1] HW × 1 , ˜Γ x ∈ R HW × 1 , ˜Γ y ∈ R HW × 1 , and ˜ Z u ∈ R HW × 1 , respectively.

Original: Organizing ˜Σ = diag ([˜Σ x ; ˜Σ y ])
Revised: Organizing ˜Σ = diag ([˜Σ x ; ˜Σ y ])

Original: ∈ R 2 HW × 2 HW and ˜Γ = concat [˜Γ x ; ˜Γ y
Revised: ∈ R 2 HW × 2 HW and ˜Γ = concat [˜Γ x ; ˜Γ y

Original: ] ∈ R 2 HW × 1 , we can write the overall expression in a compact matrix form using simple algebra as follows
Revised: ] ∈ R 2 HW × 1 , we can write the overall expression in a compact matrix form using simple algebra as follows

Original: [Equation]
Revised: [Equation]

Original: where P ∈ { 1 , − 1 } 2 HW × HW is the first-order difference operator.
Revised: where P ∈ { 1 , − 1 } 2 HW × HW is the first-order difference operator.

Original: Specifically, P is a sparse matrix with only a few elements as 1 or -1.
Revised: Specifically, P is a sparse matrix with only a few elements as 1 or -1.

Original: The i th row of P provides the first-order difference operator for the i th equation.
Revised: The i th row of P provides the first-order difference operator for the i th equation.

Original: The position of 1 and -1 indicates which pair of neighbors to be considered for the constraint.
Revised: The position of 1 and -1 indicates which pair of neighbors to be considered for the constraint.

Original: Fig.2 (b) provides a visual intuition about this matrix equation.
Revised: Fig.2 (b) provides a visual intuition about this matrix equation.

Original: Eq.(3) can be utilized to recover Z u from the predicted Γ x , Γ y , Σ x , and Σ y .
Revised: Eq.(3) can be utilized to recover Z u from the predicted Γ x , Γ y , Σ x , and Σ y .

Original: As alluded to above, we have more equations than unknowns, hence, we resort to recovering the optimal depth map
Revised: As alluded to above, we have more equations than unknowns, hence, we resort to recovering the optimal depth map

Original: [Figure]
Revised: [Figure]

Original: ( c ) U p s a m p l e a nd
Revised: ( c ) U p s a m p l e a nd

Original: [Figure]
Revised: [Figure]

Original: e f i n e
Revised: e f i n e

Original: Upsampling ...
Revised: Upsampling ...

Original: A A
Revised: A A

Original: A reArrange
Revised: A reArrange

Original: Eq.
Revised: Eq.

Original: Feature
Revised: Feature

Original: Depth Depth
Revised: Depth Depth

Original: Refine
Revised: Refine

Original: Depth
Revised: Depth

Original: Feature Feature
Revised: Feature Feature

Original: Refine Refine
Revised: Refine Refine

Original: A
Revised: A

Original: [Figure]
Revised: [Figure]

Original: ˜ Z ∗ u ∈ R HW × 1 by minimizing the following equation:
Revised: ˜ Z ∗ u ∈ R HW × 1 by minimizing the following equation:

Original: [Equation]
Revised: [Equation]

Original: Refer Appendix for the full derivation.
Revised: Refer Appendix for the full derivation.

Original: The closed-form solution can be written as follows:
Revised: The closed-form solution can be written as follows:

Original: [Equation]
Revised: [Equation]

Original: Denote K ˜Σ ≜ ( P T ˜Σ 2 P ) − 1 P T ˜Σ 2 in Eq.(5), we write overall equation as ˜ Z ∗ u = K ˜Σ ˜Γ .
Revised: Denote K ˜Σ ≜ ( P T ˜Σ 2 P ) − 1 P T ˜Σ 2 in Eq.(5), we write overall equation as ˜ Z ∗ u = K ˜Σ ˜Γ .

Original: Next, we describe the overall network architecture.
Revised: Next, we describe the overall network architecture.

Original: 3. O VERALL N ETWORK A RCHITECTURE
Revised: 3. O VERALL N ETWORK A RCHITECTURE

Original: Our overall network architecture is composed of four main modules as follows.
Revised: Our overall network architecture is composed of four main modules as follows.

Original: (a) Encoder.
Revised: (a) Encoder.

Original: Given an input image, the encoder computes the hierarchical feature maps through a series of stages.
Revised: Given an input image, the encoder computes the hierarchical feature maps through a series of stages.

Original: To be precise, our encoder has four stages.
Revised: To be precise, our encoder has four stages.

Original: Each stage contains transformer blocks (Liu et al., 2021b).
Revised: Each stage contains transformer blocks (Liu et al., 2021b).

Original: At the end of each stage, we collect the final feature map
Revised: At the end of each stage, we collect the final feature map

Original: as the output of the encoder resulting in the encoded feature maps with strides 4, 8, 16, and 32, respectively.
Revised: as the output of the encoder resulting in the encoded feature maps with strides 4, 8, 16, and 32, respectively.

Original: Our encoder module is inspired by Liu et al.
Revised: Our encoder module is inspired by Liu et al.

Original: (2021b), a recent state-of-the-art transformer network design.
Revised: (2021b), a recent state-of-the-art transformer network design.

Original: We use it as our backbone by removing the final global pooling layer and fully connected layer.
Revised: We use it as our backbone by removing the final global pooling layer and fully connected layer.

Original: (b) Variational Layer (V-Layer).
Revised: (b) Variational Layer (V-Layer).

Original: The goal of this layer is to compute a map from encoded feature maps to unscaled depth map, which adheres to the first-order variational constraint.
Revised: The goal of this layer is to compute a map from encoded feature maps to unscaled depth map, which adheres to the first-order variational constraint.

Original: As of V-layer, we feed the feature maps of strides 16 and 32 as input which is the output of the encoder.
Revised: As of V-layer, we feed the feature maps of strides 16 and 32 as input which is the output of the encoder.

Original: Since these features are at different resolutions, we up-sample the feature map of stride 32 to stride 16 via bi-linear interpolation and concatenate to construct I Φ ∈ R C × H × W , where ( H, W, C ) symbolizing the height, width, and the number of channels, respectively.
Revised: Since these features are at different resolutions, we upsample the feature map of stride 32 to stride 16 via bi-linear interpolation and concatenate to construct I Φ ∈ R C × H × W , where ( H, W, C ) symbolizing the height, width, and the number of channels, respectively.

Original: Note that H, W is not the same as the original resolution of the ground-truth depth and images.
Revised: Note that H, W is not the same as the original resolution of the ground-truth depth and images.

Original: We use of two convolutional layers on I Φ to predict the depth gradient and corresponding weight for each pixel as follows:
Revised: We use of two convolutional layers on I Φ to predict the depth gradient and corresponding weight for each pixel as follows:

Original: [Equation]
Revised: [Equation]

Original: where, f ( I Φ ; θ ) denotes the convolutional layers with parameters θ .
Revised: where, f ( I Φ ; θ ) denotes the convolutional layers with parameters θ .

Original: The predicted depth gradients Γ x and Γ y are observed to be more accurate at smooth surface than at boundaries.
Revised: The predicted depth gradients Γ x and Γ y are observed to be more accurate at smooth surface than at boundaries.

Original: This brings us again to the point made above that we must take care of which first-order constraint must be included and discarded during regression.
Revised: This brings us again to the point made above that we must take care of which first-order constraint must be included and discarded during regression.

Original: Using the Eq.(6) prediction, we construct the variational constraint Eq.(3), and obtain the unscaled depth map following Eq.(5).
Revised: Using the Eq.(6) prediction, we construct the variational constraint Eq.(3), and obtain the unscaled depth map following Eq.(5).

Original: The resulting depth map has a resolution of 1/16 to the original image, which is later upsampled to the appropriate dimension.
Revised: The resulting depth map has a resolution of 1/16 to the original image, which is later upsampled to the appropriate resolution.

Original: To capture more scene features, we generate multiple channels (denoted as S ) of { Γ x , Γ y , Σ x , Σ y } using Eq.(6).
Revised: To capture more scene features, we generate multiple channels (denoted as S ) of { Γ x , Γ y , Σ x , Σ y } using Eq.(6).

Original: As a result, we have a group of depth maps stacked along the channel dimension.
Revised: As a result, we have a group of depth maps stacked along the channel dimension.

Original: For a feature map with spatial resolution H × W , our V-layer has a complexity of O
Revised: For a feature map with spatial resolution H × W , our V-layer has a complexity of O

Original: ( H 3 W 3 ) .
Revised: ( H 3 W 3 ) .

Original: To overcome complexity issue, we perform V-layer operation on feature maps with stride 16 and then up-sample and refine the depth maps in the later stage.
Revised: To overcome complexity issue, we perform V-layer operation on feature maps with stride 16 and then upsample and refine the depth maps in the later stage.

Original: The V-layer pipeline is shown in Fig.3(b).
Revised: The V-layer pipeline is shown in Fig.3(b).

Original: (c) Upsample and Refine.
Revised: (c) Upsample and Refine.

Original: This module upsamples and refines the input depth map via encoded features at a given depth map resolution.
Revised: This module upsamples and refines the input depth map via encoded features at a given depth map resolution.

Original: To this end, we perform refinement at three different resolutions in a hierarchical manner.
Revised: To this end, we perform refinement at three different resolutions in a hierarchical manner.

Original: Given the V-layer depth map at 1/16 resolution, we first refine the depth via encoded features at this resolution.
Revised: Given the V-layer depth map at 1/16 resolution, we first refine the depth via encoded features at this resolution.

Original: Concretely, this refinement is done using the following set of operations.
Revised: Concretely, this refinement is done using the following set of operations.

Original: (1) concatenate the feature map and the depth map; (2) use one convolutional layer with ReLU activation to fuse the feature and depth information; and (3) predict refined feature and depth map via a convolutional layer.
Revised: (1) concatenate the feature map and the depth map; (2) use one convolutional layer with ReLU activation to fuse the feature and depth information; and (3) predict refined feature and depth map via a convolutional layer.

Original: Later, the refined feature and depth map are upsampled and fed into 1/8 for later refinement using the same above set of operations.
Revised: Later, the refined feature and depth map are upsampled and fed into 1/8 for later refinement using the same above set of operations.

Original: Finally, the exact is done at 1/ resolution.
Revised: Finally, the exact is done at 1/ resolution.

Original: Note that these steps are performed in a sequel.
Revised: Note that these steps are performed in a sequel.

Original: At the end of this module, we have a depth map of 1/4 of the actual resolution.
Revised: At the end of this module, we have a depth map of 1/4 of the actual resolution.

Original: The upsample and refine procedure are shown in Fig.3(c).
Revised: The upsample and refine procedure are shown in Fig.3(c).

Original: (d) Metric Layer.
Revised: (d) Metric Layer.

Original: We must infer the global scene scale and shift to recover the metric depth.
Revised: We must infer the global scene scale and shift to recover the metric depth.

Original: For this, we perform global max pooling on the encoded feature map of stride 32.
Revised: For this, we perform global max pooling on the encoded feature map of stride 32.

Original: The resulting vector is fed into a stack of fully connected layers to regress the two scalars, i.e., one representing the scale and while other representing the shift.
Revised: The resulting vector is fed into a stack of fully connected layers to regress the two scalars, i.e., one representing the scale and while other representing the shift.

Original: Using the feature map of stride 32 is motivated by the observation that we have a much richer global scene context using it than at higher depth resolution.
Revised: Using the feature map of stride 32 is motivated by the observation that we have a much richer global scene context using it than at higher depth resolution.

Original: It also provides a good compromise between computational complexity and accuracy.
Revised: It also provides a good compromise between computational complexity and accuracy.

Original: 3. L OSS F UNCTION
Revised: 3. L OSS F UNCTION

Original: Depth Loss.
Revised: Depth Loss.

Original: It estimates the scale-invariant difference between the ground-truth depth and prediction at train time (Eigen et al., 2014).
Revised: It estimates the scale-invariant difference between the ground-truth depth and prediction at train time (Eigen et al., 2014).

Original: The difference is computed by upsampling the predicted depth map to the same resolution as the ground truth via bi-linear interpolation.
Revised: The difference is computed by upsampling the predicted depth map to the same resolution as the ground truth via bi-linear interpolation.

Original: Denoting the predicted and ground-truth depth as ˆ Z ∈ R m × n , Z gt ∈ R m × n we compute the depth loss as follows
Revised: Denoting the predicted and ground-truth depth as ˆ Z ∈ R m × n , Z gt ∈ R m × n we compute the depth loss as follows

Original: [Equation]
Revised: [Equation]

Original: Here, N is the number of positions with valid measurements and α ∈ [0 , 1] is a hyper-parameter.
Revised: Here, N is the number of positions with valid measurements and α ∈ [0 , 1] is a hyper-parameter.

Original: Note that the above loss is used for valid measurements only.
Revised: Note that the above loss is used for valid measurements only.

Original: Variational Loss.
Revised: Variational Loss.

Original: We define this loss using the output of V-layer.
Revised: We define this loss using the output of V-layer.

Original: Suppose the ground-truth depth map to be Z gt ∈
Revised: Suppose the ground-truth depth map to be Z gt ∈

Original: R m × n and the predicted depth map for S channels as Z u ∈ R S × H × W .
Revised: R m × n and the predicted depth map for S channels as Z u ∈ R S × H × W .

Original: Since the depth resolution is not same at this layer, we downsample the ground truth.
Revised: Since the depth resolution is not same at this layer, we downsample the ground truth.

Original: It is observed via empirical study that low-resolution depth map in fact help capture the first-order variational loss among distant neighbors.
Revised: It is observed via empirical study that low-resolution depth map in fact help capture the first-order variational loss among distant neighbors.

Original: Accordingly, we downsample the Z gt instead of upsamping Z u .
Revised: Accordingly, we downsample the Z gt instead of upsamping Z u .

Original: We downsample Z gt denoted as Q gt ∈ R H × W by random pooling operation, i.e., we randomly select a location where we have a valid measurement since ground-truth data may have pixels with no depth values.
Revised: We downsample Z gt denoted as Q gt ∈ R H × W by random pooling operation, i.e., we randomly select a location where we have a valid measurement since ground-truth data may have pixels with no depth values.

Original: The coordinates of selected location in Z gt (cid:55)→ Z u ∈ R S × H × W and the corresponding depth value is put in ˆ Q ∈ R S × H × W via bi-linear interpolation.
Revised: The coordinates of selected location in Z gt (cid:55)→ Z u ∈ R S × H × W and the corresponding depth value is put in ˆ Q ∈ R S × H × W via bi-linear interpolation.

Original: We compute the variational loss as
Revised: We compute the variational loss as

Original: [Equation]
Revised: [Equation]

Original: where N ′ is the number of positions having valid measurements, ∇ symbolises the first-order difference operator, and Conv refers to the convolutional layer.
Revised: where N ′ is the number of positions having valid measurements, ∇ symbolises the first-order difference operator, and Conv refers to the convolutional layer.

Original: Here, we use the Conv layer to fuse S depth maps into a single depth map and also to compute its horizontal and vertical gradient.
Revised: Here, we use the Conv layer to fuse S depth maps into a single depth map and also to compute its horizontal and vertical gradient.

Original: Total Loss.
Revised: Total Loss.

Original: We define the total loss as the sum of the depth loss and the variational loss i.e.,
Revised: We define the total loss as the sum of the depth loss and the variational loss i.e.,

Original: L = L depth + λ L var , where λ is the regularization parameter set to 0.1 for all our experiments.
Revised: L = L depth + λ L var , where λ is the regularization parameter set to 0.1 for all our experiments.

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  E XPERIMENTS AND R ESULTS
Revised:  E XPERIMENTS AND R ESULTS

Original: Implementation Details We implemented our method in PyTorch 1.7.1 (Python 3.8) with CUDA 11.0.
Revised: Implementation Details We implemented our method in PyTorch 1.7.1 (Python 3.8) with CUDA 11.0.

Original: The software is evaluated on a computing machine with Quadro-RTX-6000 GPU.
Revised: The software is evaluated on a computing machine with Quadro-RTX-6000 GPU.

Original: Datasets.
Revised: Datasets.

Original: We performed experiments on three benchmark datasets namely NYU Depth V2 (Silberman et al., 2012), KITTI (Geiger et al., 2012), and SUN
Revised: We performed experiments on three benchmark datasets namely NYU Depth V2 (Silberman et al., 2012), KITTI (Geiger et al., 2012), and SUN

Original: Training Details.
Revised: Training Details.

Original: We use (Liu et al., 2021b) network as our backbone, which is pre-trained on
Revised: We use (Liu et al., 2021b) network as our backbone, which is pre-trained on

Original: ImageNet (Deng et al., 2009).
Revised: ImageNet (Deng et al., 2009).

Original: We use the Adam optimizer (Kingma & Ba, 2014) without weight decay.
Revised: We use the Adam optimizer (Kingma & Ba, 2014) without weight decay.

Original: We decrease the learning rate from 3 e − 5 to 1 e − 5 by the cosine annealing scheduler.
Revised: We decrease the learning rate from 3 e − 5 to 1 e − 5 by the cosine annealing scheduler.

Original: To avoid over-fitting, we augment the images by horizontal flipping.
Revised: To avoid over-fitting, we augment the images by horizontal flipping.

Original: For KITTI (Geiger et al., 2012), the model is trained for 10 epochs for the official split and 20 epochs for the Eigen split (Eigen et al., 2014).
Revised: For KITTI (Geiger et al., 2012), the model is trained for 10 epochs for the official split and 20 epochs for the Eigen split (Eigen et al., 2014).

Original: For NYU Depth V2 (Silberman et al., 2012), the model is trained for 20 epochs.
Revised: For NYU Depth V2 (Silberman et al., 2012), the model is trained for 20 epochs.

Original: Evaluation Metrics.
Revised: Evaluation Metrics.

Original: We report statistical results on popular evaluation metrics such as square root of the Scale Invariant Logarithmic error ( SILog ), Relative Squared error ( Sq Rel ), Relative Absolute
Revised: We report statistical results on popular evaluation metrics such as square root of the Scale Invariant Logarithmic error ( SILog ), Relative Squared error ( Sq Rel ), Relative Absolute

Original: Error ( Abs Rel ), Root Mean Squared error ( RMS ), and threshold accuracy.
Revised: Error ( Abs Rel ), Root Mean Squared error ( RMS ), and threshold accuracy.

Original: Mathematical definition related to each one of them is provided in the Appendix.
Revised: Mathematical definition related to each one of them is provided in the Appendix.

Original: 4. C OMPARISON TO S TATE OF THE A RT
Revised: 4. C OMPARISON TO S TATE OF THE A RT

Original: Tab.(1), Tab.(2), Tab.(3), and Tab.(4) provide statistical comparison results with the competing methods on NYU Depth V2, KITTI official split, KITTI Eigen split, and SUN RGB-D, respectively.
Revised: Tab.(1), Tab.(2), Tab.(3), and Tab.(4) provide statistical comparison results with the competing methods on NYU Depth V2, KITTI official split, KITTI Eigen split, and SUN RGB-D, respectively.

Original: Our proposed approach shows the best results for all the evaluation metrics.
Revised: Our proposed approach shows the best results for all the evaluation metrics.

Original: Particularly on the NYU test set, we reduce the SILog error from the previous best result, 9.102 to 8.198, and increase δ 1 from 0.922 to 0.937.
Revised: Particularly on the NYU test set, we reduce the SILog error from the previous best result, 9.102 to 8.198, and increase δ 1 from 0.922 to 0.937.

Original: More qualitative results including V-layer output are presented in the Appendix.
Revised: More qualitative results including V-layer output are presented in the Appendix.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: For the SUN RGB-D test set, all competing models, including ours, are trained on the NYU Depth
Revised: For the SUN RGB-D test set, all competing models, including ours, are trained on the NYU Depth

Original: V2 training set (Silberman et al., 2012) without fine-tuning on the SUN RGB-D.
Revised: V2 training set (Silberman et al., 2012) without fine-tuning on the SUN RGB-D.

Original: In addition, we align the predictions from all the models with the ground truth by a scale and shift following Ranftl et al.
Revised: In addition, we align the predictions from all the models with the ground truth by a scale and shift following Ranftl et al.

Original: Tab.(4) results show our method’s better generalization capability than other approaches.
Revised: Tab.(4) results show our method’s better generalization capability than other approaches.

Original: Extensive visual results are provided in the Appendix and supplementary video.
Revised: Extensive visual results are provided in the Appendix and supplementary video.

Original: 4. A BLATION S TUDY
Revised: 4. A BLATION S TUDY

Original: All the ablation presented below is conducted on NYU Depth V2 test set (Silberman et al., 2012).
Revised: All the ablation presented below is conducted on NYU Depth V2 test set (Silberman et al., 2012).

Original: (i) Effect of V-Layer.
Revised: (i) Effect of V-Layer.

Original: To understand the benefit and outcome of our variational layer compared to other popular alternative layers in deep neural networks for this problem, we performed this ablation study.
Revised: To understand the benefit and outcome of our variational layer compared to other popular alternative layers in deep neural networks for this problem, we performed this ablation study.

Original: We replace our V-layer firstly with a convolutional layer and later with a self-attention layer.
Revised: We replace our V-layer firstly with a convolutional layer and later with a self-attention layer.

Original: Tab.(5) provides the depth prediction accuracy for this ablation.
Revised: Tab.(5) provides the depth prediction accuracy for this ablation.

Original: For each introduced layer in Tab.(5), the first and second rows show the performance of the depth map predicted with (w) and without (w/o) subsequent refinements (cf.
Revised: For each introduced layer in Tab.(5), the first and second rows show the performance of the depth map predicted with (w) and without (w/o) subsequent refinements (cf.

Original: Sec.3.2 (c)), respectively.
Revised: Sec.3.2 (c)), respectively.

Original: For the self-attention layer, we follow the ViT (Dosovitskiy et al., 2021) and set the patch size to be one as we use the feature map with stride 16.
Revised: For the self-attention layer, we follow the ViT (Dosovitskiy et al., 2021) and set the patch size to be one as we use the feature map with stride 16.

Original: We also adopt the learnable position embedding (PE) with 128 dimensions.
Revised: We also adopt the learnable position embedding (PE) with 128 dimensions.

Original: We set the number of heads to be 4 and the number of hidden units to be 512.
Revised: We set the number of heads to be 4 and the number of hidden units to be 512.

Original: As shown in Tab.(5), our V-layer indeed helps improve the accuracy of depth prediction compared to other well-known layers.
Revised: As shown in Tab.(5), our V-layer indeed helps improve the accuracy of depth prediction compared to other well-known layers.

Original: More experiments on KITTI and SUN RBG-D are provided in the Appendix.
Revised: More experiments on KITTI and SUN RBG-D are provided in the Appendix.

Original: (ii) Performance with Different Network Backbone.
Revised: (ii) Performance with Different Network Backbone.

Original: We evaluate the effects of our V-layer with different types of network backbones.
Revised: We evaluate the effects of our V-layer with different types of network backbones.

Original: For this ablation, we use Swin-Large (Liu et al., 2021b),
Revised: For this ablation, we use Swin-Large (Liu et al., 2021b),

Original: Swin-Small (Liu et al., 2021b), and ConvNeXt-Small (Liu et al., 2022).
Revised: Swin-Small (Liu et al., 2021b), and ConvNeXt-Small (Liu et al., 2022).

Original: The SILog error is shown in Fig.
Revised: The SILog error is shown in Fig.

Original: The results show that our V-layer improves the transformer and the convolutional network performance.
Revised: The results show that our V-layer improves the transformer and the convolutional network performance.

Original: An important observation is that our V-layer shows excellent improvements in depth prediction accuracy on weaker network backbones.
Revised: An important observation is that our V-layer shows excellent improvements in depth prediction accuracy on weaker network backbones.

Original: (iii) Performance with Change in the Value of S .
Revised: (iii) Performance with Change in the Value of S .

Original: For this ablation, we change the value of S in the V-layer and observe its effects (cf.
Revised: For this ablation, we change the value of S in the V-layer and observe its effects (cf.

Original: Sec.3.2 (b)).
Revised: Sec.3.2 (b)).

Original: By increasing S , we generate more channels of ˜Γ and ˜Σ which in-effect increases V-layer parameters.
Revised: By increasing S , we generate more channels of ˜Γ and ˜Σ which in-effect increases V-layer parameters.

Original: In the subsequent step, we expand the
Revised: In the subsequent step, we expand the

Original: [Table]
Revised: [Table]

Original: w
Revised: w

Original: [Table]
Revised: [Table]

Original: number of channels to 128 by a convolutional layer to use the subsequent layers as they are.
Revised: number of channels to 128 by a convolutional layer to use the subsequent layers as they are.

Original: The results are shown in Tab.(6).
Revised: The results are shown in Tab.(6).

Original: For reference, we also present the result by replacing the V-layer with a convolutional layer in the first row in Tab.(6).
Revised: For reference, we also present the result by replacing the V-layer with a convolutional layer in the first row in Tab.(6).

Original: By increasing S , we reduce the SILog error, at the price of the speed (FPS).
Revised: By increasing S , we reduce the SILog error, at the price of the speed (FPS).

Original: Yet, no real benefit is observed with S more than 16.
Revised: Yet, no real benefit is observed with S more than 16.

Original: [Figure]
Revised: [Figure]

Original: (iv) Effect of Confidence Weight Matrix & Difference Operator in V-Layer.
Revised: (iv) Effect of Confidence Weight Matrix & Difference Operator in V-Layer.

Original: For this ablation, we study the network’s depth prediction under four different settings.
Revised: For this ablation, we study the network’s depth prediction under four different settings.

Original: (a) without Vlayer and replace it with convolutional layer (b) without the confidence weight matrix (c) with learnable difference operator and (d) our full model.
Revised: (a) without Vlayer and replace it with convolutional layer (b) without the confidence weight matrix (c) with learnable difference operator and (d) our full model.

Original: The depth prediction accuracy observed under these settings is provided in Tab.(7).
Revised: The depth prediction accuracy observed under these settings is provided in Tab.(7).

Original: Clearly, our full model has better accuracy.
Revised: Clearly, our full model has better accuracy.

Original: An important empirical observation, we made during this test is when we keep P learnable V-layer has more learnable parameters, the performance becomes worse than with fixed difference operator.
Revised: An important empirical observation, we made during this test is when we keep P learnable V-layer has more learnable parameters, the performance becomes worse than with fixed difference operator.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: the difference operator P .
Revised: the difference operator P .

Original: [Table]
Revised: [Table]

Original: 4. N ETWORK PROCESSING TIME & P ARAMETERS
Revised: 4. N ETWORK PROCESSING TIME & P ARAMETERS

Original: We compared our method’s inference time and the number of model parameters to the AdaBins (Bhat et al., 2021) and the NeWCRFs (Yuan et al., 2022).
Revised: We compared our method’s inference time and the number of model parameters to the AdaBins (Bhat et al., 2021) and the NeWCRFs (Yuan et al., 2022).

Original: The inference time is measured on the NYU Depth V2 test set with batch size 1.
Revised: The inference time is measured on the NYU Depth V2 test set with batch size 1.

Original: We have removed the ensemble tricks in AdaBins and NeWCRFs for an unbiased evaluation, resulting in a slight increase in SILog error as compared to Tab.(1) statistics.
Revised: We have removed the ensemble tricks in AdaBins and NeWCRFs for an unbiased evaluation, resulting in a slight increase in SILog error as compared to Tab.(1) statistics.

Original: As is shown in Tab.(8), our method is faster and better than AdaBins and NeWCRFs using Swin-Small backbone.
Revised: As is shown in Tab.(8), our method is faster and better than AdaBins and NeWCRFs using Swin-Small backbone.

Original: With the same backbone as the NeWCRFs, i
Revised: With the same backbone as the NeWCRFs, i

Original: [Table]
Revised: [Table]

Original: e.,
Revised: e.,

Original: Swin-Large, we achieve much better depth prediction results.
Revised: Swin-Large, we achieve much better depth prediction results.

Original: Hence, our method with Swin-Small backbone provides a better balance between accuracy, speed and memory foot-print.
Revised: Hence, our method with Swin-Small backbone provides a better balance between accuracy, speed and memory foot-print.

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In conclusion, a simple and effective approach for inferring scene depth from a single image is introduced.
Revised: In conclusion, a simple and effective approach for inferring scene depth from a single image is introduced.

Original: The proposed SIDP approach is shown to better exploit the rigid scene prior, which is generally overlooked by the existing neural network-based methods.
Revised: The proposed SIDP approach is shown to better exploit the rigid scene prior, which is generally overlooked by the existing neural network-based methods.

Original: Our approach does not make explicit assumptions about the scene other than the scene gradient regularity, which holds for typical indoor or outdoor scenes.
Revised: Our approach does not make explicit assumptions about the scene other than the scene gradient regularity, which holds for typical indoor or outdoor scenes.

Original: When tested on popular benchmark datasets, our method shows significantly better results than the prior art, both qualitatively and quantitatively.
Revised: When tested on popular benchmark datasets, our method shows significantly better results than the prior art, both qualitatively and quantitatively.

Original:  A CKNOWLEDGMENT
Revised:  A CKNOWLEDGMENT

Original: This work was partly supported by ETH General Fund (OK), Chinese Scholarship Council (CSC), and The Alexander von Humboldt Foundation.
Revised: This work was partly supported by ETH General Fund (OK), Chinese Scholarship Council (CSC), and The Alexander von Humboldt Foundation.

Original: A A PPENDIX A. T RAINING D ETAILS
Revised: A A PPENDIX A. T RAINING D ETAILS

Original: We implement our framework in PyTorch (Paszke et al., 2019).
Revised: We implement our framework in PyTorch (Paszke et al., 2019).

Original: We adopt the Swin-Large (Liu et al., 2021b) as our backbone to conduct ablation experiments and compare with the state of the arts.
Revised: We adopt the Swin-Large (Liu et al., 2021b) as our backbone to conduct ablation experiments and compare with the state of the arts.

Original: And the backbone is pre-trained on ImageNet-22K (Deng et al., 2009) by image classification.
Revised: And the backbone is pre-trained on ImageNet-22K (Deng et al., 2009) by image classification.

Original: For training, we use the Adam optimizer (Kingma & Ba, 2014) without weight decay.
Revised: For training, we use the Adam optimizer (Kingma & Ba, 2014) without weight decay.

Original: We decrease the learning rate from 3 e − 5 to 1 e − 5 by the cosine annealing scheduler.
Revised: We decrease the learning rate from 3 e − 5 to 1 e − 5 by the cosine annealing scheduler.

Original: To avoid over-fitting, we augment the images by horizontal flipping.
Revised: To avoid over-fitting, we augment the images by horizontal flipping.

Original: For both train and test, we keep the resolution of images to be 352 × 1216 in KITTI (Geiger et al., 2012) and 480 × 640 in both NYU Depth V2 (Silberman et al., 2012) and SUN RGB-D (Song et al., 2015).
Revised: For both train and test, we keep the resolution of images to be 352 × 1216 in KITTI (Geiger et al., 2012) and 480 × 640 in both NYU Depth V2 (Silberman et al., 2012) and SUN RGB-D (Song et al., 2015).

Original: In KITTI the model is trained for 10 epochs for the official split and 20 epochs for the Eigen split (Eigen et al., 2014).
Revised: In KITTI the model is trained for 10 epochs for the official split and 20 epochs for the Eigen split (Eigen et al., 2014).

Original: In NYU Depth V2, the model is trained for 20 epochs.
Revised: In NYU Depth V2, the model is trained for 20 epochs.

Original: We set the batch size to be 4 and 8 respectively for ablation experiments and comparison to the state of the arts.
Revised: We set the batch size to be 4 and 8 respectively for ablation experiments and comparison to the state of the arts.

Original: A. E VALUATION M ETRICS
Revised: A. E VALUATION M ETRICS

Original: Suppose the predicted and ground-truth depth to be ˆ Z ∈ R m × n and Z gt ∈
Revised: Suppose the predicted and ground-truth depth to be ˆ Z ∈ R m × n and Z gt ∈

Original: R m × n , respectively, and the number of valid pixels to be N .
Revised: R m × n , respectively, and the number of valid pixels to be N .

Original: We follow the existing methods (Yuan et al., 2022) and utilize the following measures for quantitative evaluation:
Revised: We follow the existing methods (Yuan et al., 2022) and utilize the following measures for quantitative evaluation:

Original: • square root of the Scale Invariant Logarithmic error ( SILog ): 1 N (cid:80) i,j ( e i,j ) 2 − 1 N 2 ( (cid:80) i,j e i,j ) 2 , where e i,j = log ˆ
Revised: • square root of the Scale Invariant Logarithmic error ( SILog ): 1 N (cid:80) i,j ( e i,j ) 2 − 1 N 2 ( (cid:80) i,j e i,j ) 2 , where e i,j = log ˆ

Original: Z i,j − log Z i,jgt ; • Relative Squared error ( Sq Rel ): 1 N (cid:80) i,j
Revised: Z i,j − log Z i,jgt ; • Relative Squared error ( Sq Rel ): 1 N (cid:80) i,j

Original: ( ˆ Z i,j − Z i,jgt ) 2 /Z
Revised: ( ˆ Z i,j − Z i,jgt ) 2 /Z

Original: i,jgt ; • Relative Absolute Error ( Abs Rel ): 1 N (cid:80) i,j | ˆ Z i,j − Z i,jgt | /Z
Revised: i,jgt ; • Relative Absolute Error ( Abs Rel ): 1 N (cid:80) i,j | ˆ Z i,j − Z i,jgt | /Z

Original: i,jgt ; • Root Mean Squared error ( RMS ): 1 N (cid:113)(cid:80) i,j
Revised: i,jgt ; • Root Mean Squared error ( RMS ): 1 N (cid:113)(cid:80) i,j

Original: ( ˆ Z i,j − Z i,jgt ) 2 ; • Root Mean Squared Logarithmic error ( RMS log ): 1 N (cid:113)(cid:80) i,j ( e i,j ) 2 ; • threshold accuracy ( δ k ): percentage of ˆ Z i,j s.t. max( ˆ Z i,j Z i,j gt , Z i,jgt ˆ Z i,j ) < 1 .
Revised: ( ˆ Z i,j − Z i,jgt ) 2 ; • Root Mean Squared Logarithmic error ( RMS log ): 1 N (cid:113)(cid:80) i,j ( e i,j ) 2 ; • threshold accuracy ( δ k ): percentage of ˆ Z i,j s.t. max( ˆ Z i,j Z i,j gt , Z i,jgt ˆ Z i,j ) < 1 .

Original: 25 k .
Revised: 25 k .

Original: A. D ERIVATION OF THE V ARIATIONAL C ONSTRAINT
Revised: A. D ERIVATION OF THE V ARIATIONAL C ONSTRAINT

Original: In this part, we introduce the derivation of the variational constraint in more detail.
Revised: In this part, we introduce the derivation of the variational constraint in more detail.

Original: Firstly, we present the detailed form of the difference operator P , the confidence weight matrix ˜Σ , the difference vector ˜Γ and the depth vector ˜ Z u , when the depth map Z u is at resolution 2 × 2 .
Revised: Firstly, we present the detailed form of the difference operator P , the confidence weight matrix ˜Σ , the difference vector ˜Γ and the depth vector ˜ Z u , when the depth map Z u is at resolution 2 × 2 .

Original: Secondly we show the derivation of the optimal solution ˜
Revised: Secondly we show the derivation of the optimal solution ˜

Original: Z ∗ u .
Revised: Z ∗ u .

Original: Overall Matrix Form.
Revised: Overall Matrix Form.

Original: Suppose the unscaled depth map to be Z u ∈ R 2 × 2 , the difference map to be Γ x ∈ R 2 × 2 and Γ y ∈ R 2 × 2 , and the corresponding confidence weight map to be Σ x ∈ [0 , 1] 2 ×and Σ y ∈ [0 , 1] 2 × 2 .
Revised: Suppose the unscaled depth map to be Z u ∈ R 2 × 2 , the difference map to be Γ x ∈ R 2 × 2 and Γ y ∈ R 2 × 2 , and the corresponding confidence weight map to be Σ x ∈ [0 , 1] 2 ×and Σ y ∈ [0 , 1] 2 × 2 .

Original: We first reorganize the elements in Γ x , Γ y , Σ x , Σ y to construct the difference vector ˜Γ ∈ R 8 × 1 and the confidence weight matrix ˜Σ ∈ R 8 × 8 :
Revised: We first reorganize the elements in Γ x , Γ y , Σ x , Σ y to construct the difference vector ˜Γ ∈ R 8 × 1 and the confidence weight matrix ˜Σ ∈ R 8 × 8 :

Original: [Equation]
Revised: [Equation]

Original:  
Revised:  

Original: [Equation]
Revised: [Equation]

Original:  
Revised:  

Original: [Equation]
Revised: [Equation]

Original: =   Σ
Revised: =   Σ

Original: [Equation]
Revised: [Equation]

Original: 1 x        Σ 1 , 2 x        Σ 2 , 1 x        Σ 2 , 2 x        Σ 1 , 1 y        Σ 1 , 2 y        Σ 2 , 1 y        Σ 2 , 2 y   .
Revised: 1 x        Σ 1 , 2 x        Σ 2 , 1 x        Σ 2 , 2 x        Σ 1 , 1 y        Σ 1 , 2 y        Σ 2 , 1 y        Σ 2 , 2 y   .

Original: Next, we apply the difference operator P ∈ R 8 × 4 :
Revised: Next, we apply the difference operator P ∈ R 8 × 4 :

Original: P = 
Revised: P = 

Original:  −         − 1    −    −            on the depth vector ˜ Z u ∈ R 4 × 1 :
Revised:  −         − 1    −    −            on the depth vector ˜ Z u ∈ R 4 × 1 :

Original: [Equation]
Revised: [Equation]

Original: =  
Revised: =  

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Z
Revised: Z

Original: [Equation]
Revised: [Equation]

Original: Z
Revised: Z

Original: [Equation]
Revised: [Equation]

Original: Z
Revised: Z

Original: [Equation]
Revised: [Equation]

Original:   .
Revised:   .

Original: Finally, we construct the following constraint equation:
Revised: Finally, we construct the following constraint equation:

Original: [Equation]
Revised: [Equation]

Original: In the above equation, each row represents a weighted constraint for the first-order difference along the x -axis:
Revised: In the above equation, each row represents a weighted constraint for the first-order difference along the x -axis:

Original: [Equation]
Revised: [Equation]

Original: or along the y -axis:
Revised: or along the y -axis:

Original: [Equation]
Revised: [Equation]

Original: where ( Z i,j +1 u − Z i,ju )
Revised: where ( Z i,j +1 u − Z i,ju )

Original: =
Revised: =

Original: Γ i,jx and ( Z i +1 ,j u − Z i,ju ) =
Revised: Γ i,jx and ( Z i +1 ,j u − Z i,ju ) =

Original: Γ i,jy are the variational constraints for the first-order difference, while Σ i,jx and Σ i,jy are the confidence weights for the constraints.
Revised: Γ i,jy are the variational constraints for the first-order difference, while Σ i,jx and Σ i,jy are the confidence weights for the constraints.

Original: Optimal Solution.
Revised: Optimal Solution.

Original: Given ˜Σ and ˜Γ , we search for the optimal depth vector ˜ Z ∗ u by minimizing the residual of Eq.(9):
Revised: Given ˜Σ and ˜Γ , we search for the optimal depth vector ˜ Z ∗ u by minimizing the residual of Eq.(9):

Original: [Equation]
Revised: [Equation]

Original: Firstly, the objective function, || ˜Σ(
Revised: Firstly, the objective function, || ˜Σ(

Original: P ˜
Revised: P ˜

Original: Z u − ˜Γ) || 2 , is convex with respect to ˜ Z u .
Revised: Z u − ˜Γ) || 2 , is convex with respect to ˜ Z u .

Original: Because || · || 2 is a convex function.
Revised: Because || · || 2 is a convex function.

Original: A composition of || · || 2 and the affine function , ˜Σ( P ˜
Revised: A composition of || · || 2 and the affine function , ˜Σ( P ˜

Original: Z
Revised: Z

Original: u − ˜Γ) , is still convex.
Revised: u − ˜Γ) , is still convex.

Original: Secondly, the optimal solution of a convex function can be found at where the first derivative is zero.
Revised: Secondly, the optimal solution of a convex function can be found at where the first derivative is zero.

Original: Thereby we obtain the final solution ˜
Revised: Thereby we obtain the final solution ˜

Original: Z ∗ u =
Revised: Z ∗ u =

Original: ( P T ˜ Σ 2 P ) − 1 P T ˜ Σ 2 ˜ Γ .
Revised: ( P T ˜ Σ 2 P ) − 1 P T ˜ Σ 2 ˜ Γ .

Original: A. N ETWORK S TRUCTURE D ETAILS
Revised: A. N ETWORK S TRUCTURE D ETAILS

Original: In this part, we introduce the detailed structure of our network.
Revised: In this part, we introduce the detailed structure of our network.

Original: In general we set the kernel size of convolutional layers to be 3 unless otherwise stated.
Revised: In general we set the kernel size of convolutional layers to be 3 unless otherwise stated.

Original: (a) Encoder.
Revised: (a) Encoder.

Original: We adopt the Swin-Large (Liu et al., 2021b) as our backbone.
Revised: We adopt the Swin-Large (Liu et al., 2021b) as our backbone.

Original: The network first divides the image into patches each of size 4 × 4 , and embeds each patch into a 96 -dimensional vector.
Revised: The network first divides the image into patches each of size 4 × 4 , and embeds each patch into a 96 -dimensional vector.

Original: The above procedure is implemented by a convolutional layer with kernel size 4 and stride 4.
Revised: The above procedure is implemented by a convolutional layer with kernel size 4 and stride 4.

Original: Then there are 4 transformation stages to be applied to the embedded vector.
Revised: Then there are 4 transformation stages to be applied to the embedded vector.

Original: The first, second, third and forth stages include 2 , 2 , 18 , 2 blocks, respectively.
Revised: The first, second, third and forth stages include 2 , 2 , 18 , 2 blocks, respectively.

Original: More specifically, each block will divide the feature map into non-overlapped windows of size 12 × 12 , and compute new features within each window following the transformer (Vaswani et al., 2017).
Revised: More specifically, each block will divide the feature map into non-overlapped windows of size 12 × 12 , and compute new features within each window following the transformer (Vaswani et al., 2017).

Original: The feature channels in the 4 stages are 192, 384, 768 and 1536, respectively, and the number of heads are 6, 12, 24, and 48, respectively.
Revised: The feature channels in the 4 stages are 192, 384, 768 and 1536, respectively, and the number of heads are 6, 12, 24, and 48, respectively.

Original: In the end of each stage, there will be a downsampling operation to reduce the resolution of the feature map by 2.
Revised: In the end of each stage, there will be a downsampling operation to reduce the resolution of the feature map by 2.

Original: We collect the feature map before the downsampling operation as the output of the stage.
Revised: We collect the feature map before the downsampling operation as the output of the stage.

Original: In the end, the strides of output feature maps are 4, 8, 16, and 32 respectively.
Revised: In the end, the strides of output feature maps are 4, 8, 16, and 32 respectively.

Original: And the channels are 192, 384, 768, and 1536, respectively.
Revised: And the channels are 192, 384, 768, and 1536, respectively.

Original: (b) Upsampling
Revised: (b) Upsampling

Original: >
Revised: >

Original: Given the feature maps of strides 32 and 16 from
Revised: Given the feature maps of strides 32 and 16 from

Original: (a
Revised: (a

Original: ) , we first upsample the feature map of stride 32 to stride 16 via bi-linear interpolation.
Revised: ) , we first upsample the feature map of stride 32 to stride 16 via bi-linear interpolation.

Original: Then, we concatenate the feature maps and obtain a new feature map of stride 16 and channels 1536 + 768 = 2304 .
Revised: Then, we concatenate the feature maps and obtain a new feature map of stride 16 and channels 1536 + 768 = 2304 .

Original: We apply a convolutional layer that has 2304 input channels, 2304 output channels, and 4 groups, to fuse the information.
Revised: We apply a convolutional layer that has 2304 input channels, 2304 output channels, and 4 groups, to fuse the information.

Original: We also append an instance normalization layer (Ulyanov et al., 2016) and a LeakyReLU activation function.
Revised: We also append an instance normalization layer (Ulyanov et al., 2016) and a LeakyReLU activation function.

Original: Next, we apply another convolutional layer with 2304 input channels, 512 output channels to compress the feature channels.
Revised: Next, we apply another convolutional layer with 2304 input channels, 512 output channels to compress the feature channels.

Original: Again we append an instance normalization layer and the LeakyReLU activation function.
Revised: Again we append an instance normalization layer and the LeakyReLU activation function.

Original: In the end, we add a skip connection between the result feature map and the previous concatenated feature map by addition operation.
Revised: In the end, we add a skip connection between the result feature map and the previous concatenated feature map by addition operation.

Original: The concatenated feature map will be transformed to consistent number of channels in advance by a convolutional layer with 2304 input channels and
Revised: The concatenated feature map will be transformed to consistent number of channels in advance by a convolutional layer with 2304 input channels and

Original: output
Revised: output

Original: channels.
Revised: channels.

Original: The final feature map has stride 16 and 512 channels.
Revised: The final feature map has stride 16 and 512 channels.

Original: (c) V-Layer.
Revised: (c) V-Layer.

Original: The V-layer takes the output feature map from (b) as input.
Revised: The V-layer takes the output feature map from (b) as input.

Original: We first utilize a convo-
Revised: We first utilize a convo-

Original: lutional layer with 512 input channels and 512 output channels to transform the feature map into a more appropriate hidden space.
Revised: lutional layer with 512 input channels and 512 output channels to transform the feature map into a more appropriate hidden space.

Original: The feature is transformed by the LeakyReLU activation function.
Revised: The feature is transformed by the LeakyReLU activation function.

Original: Then we utilize another convolutional layer with 512 input channels and 2 × 16 = 32 output channels to predict the gradients along the horizontal and vertical axis, respectively.
Revised: Then we utilize another convolutional layer with 512 input channels and 2 × 16 = 32 output channels to predict the gradients along the horizontal and vertical axis, respectively.

Original: The numberrepresents that we predict 16 channels of horizontal gradients and 16 channels of vertical gradients respectively, where each channel is expected to capture different information in scenes.
Revised: The numberrepresents that we predict 16 channels of horizontal gradients and 16 channels of vertical gradients respectively, where each channel is expected to capture different information in scenes.

Original: Similarly, we also use another convolutional layer with 512 input channels, 2 × 16 = 32 output channels to predict the corresponding confidence weight maps.
Revised: Similarly, we also use another convolutional layer with 512 input channels, 2 × 16 = 32 output channels to predict the corresponding confidence weight maps.

Original: The confidence weight will be transformed by the Sigmoid function.
Revised: The confidence weight will be transformed by the Sigmoid function.

Original: Then we reshape the predictions and compute the unscaled depth map for each pair of gradient and confidence weight following Eq.(5).
Revised: Then we reshape the predictions and compute the unscaled depth map for each pair of gradient and confidence weight following Eq.(5).

Original: The unscaled depth maps computed from all the pairs will be concatenated along the channel.
Revised: The unscaled depth maps computed from all the pairs will be concatenated along the channel.

Original: Thereby we obtain a depth map withchannels and stride 16.
Revised: Thereby we obtain a depth map withchannels and stride 16.

Original: We apply a group normalization on the depth map, where the 16 channels are viewed as a single group.
Revised: We apply a group normalization on the depth map, where the 16 channels are viewed as a single group.

Original: In the end, we expand the channels of the depth map into 128 by a convolutional layer.
Revised: In the end, we expand the channels of the depth map into 128 by a convolutional layer.

Original: Thereby the output of the V-layer is a depth map with 128 channels and stride 16.
Revised: Thereby the output of the V-layer is a depth map with 128 channels and stride 16.

Original: (d) Refine 16.
Revised: (d) Refine 16.

Original: We refine the feature map from (b) and the depth map from (c) .
Revised: We refine the feature map from (b) and the depth map from (c) .

Original: Specifically, we first concatenate the feature map with depth map, and obtain a new feature map with 512 + 128 =channels.
Revised: Specifically, we first concatenate the feature map with depth map, and obtain a new feature map with 512 + 128 =channels.

Original: Then we employ a convolutional layer with 640 input channels and 640 output channels to fuse the information.
Revised: Then we employ a convolutional layer with 640 input channels and 640 output channels to fuse the information.

Original: The result feature map will be further transformed by the LeakyReLU activation function.
Revised: The result feature map will be further transformed by the LeakyReLU activation function.

Original: Next a convolutional layer with 640 input channels and 128 output channels is applied to predict the refined depth map.
Revised: Next a convolutional layer with 640 input channels and 128 output channels is applied to predict the refined depth map.

Original: Similarly, we also predict the refined feature map by a convolutional layer with 640 input channels and 512 output channels.
Revised: Similarly, we also predict the refined feature map by a convolutional layer with 640 input channels and 512 output channels.

Original: (e) Upsampling 16->8.
Revised: (e) Upsampling 16->8.

Original: Here, we upsample the feature map from (d) to stride 8 via bi-linear interpolation.
Revised: Here, we upsample the feature map from (d) to stride 8 via bi-linear interpolation.

Original: To recover the information of scenes, we concatenate with the feature map from the encoder that has the same stride.
Revised: To recover the information of scenes, we concatenate with the feature map from the encoder that has the same stride.

Original: In such a way, we obtain a new feature map with 384 + 512 =channels and stride 8.
Revised: In such a way, we obtain a new feature map with 384 + 512 =channels and stride 8.

Original: Similar to (b) , we apply a convolutional layer with 896 input channels,output channels, and 4 groups to fuse the information.
Revised: Similar to (b) , we apply a convolutional layer with 896 input channels,output channels, and 4 groups to fuse the information.

Original: Then the feature map is transformed by the instance normalization layer and the LeakyReLU activation function.
Revised: Then the feature map is transformed by the instance normalization layer and the LeakyReLU activation function.

Original: Next, we apply another convolutional layer with 896 input channels and 256 output channels to compress feature channels.
Revised: Next, we apply another convolutional layer with 896 input channels and 256 output channels to compress feature channels.

Original: Again, we apply the instance normalization layer and LeakyReLU activation function.
Revised: Again, we apply the instance normalization layer and LeakyReLU activation function.

Original: Same as (b) , we add a skip connection between the result feature map and the previous concatenated feature map by addition operation.
Revised: Same as (b) , we add a skip connection between the result feature map and the previous concatenated feature map by addition operation.

Original: The concatenated feature map will be transformed to the consistent number of channels by a convolutional layer with 896 input channels and 256 output channels.
Revised: The concatenated feature map will be transformed to the consistent number of channels by a convolutional layer with 896 input channels and 256 output channels.

Original: Thereby the final feature map has stride 8 and 256 channels.
Revised: Thereby the final feature map has stride 8 and 256 channels.

Original: (f) Refine 8.
Revised: (f) Refine 8.

Original: We refine the depth map from (d) and the feature map from (e) .
Revised: We refine the depth map from (d) and the feature map from (e) .

Original: The procedure is the same as (d) , except for the number of channels in convolutional layers are adapted accordingly.
Revised: The procedure is the same as (d) , except for the number of channels in convolutional layers are adapted accordingly.

Original: Thereby we introduce the pipeline briefly.
Revised: Thereby we introduce the pipeline briefly.

Original: We first upsample the depth map from (d) into strideto concatenate with the feature map.
Revised: We first upsample the depth map from (d) into strideto concatenate with the feature map.

Original: Thereby we obtain a new feature map with 256 + 128 =channels.
Revised: Thereby we obtain a new feature map with 256 + 128 =channels.

Original: Then, we apply a convolutional layer with 384 input channels and 384 output channels to fuse the information.
Revised: Then, we apply a convolutional layer with 384 input channels and 384 output channels to fuse the information.

Original: The feature map is then transformed by LeakyReLU activation function.
Revised: The feature map is then transformed by LeakyReLU activation function.

Original: Next, we utilize a convolutional layer with 384 input channels and 128 output channels to predict the refined depth map.
Revised: Next, we utilize a convolutional layer with 384 input channels and 128 output channels to predict the refined depth map.

Original: Similarly, another convolutional layer with 384 input channels and 256 output channels is applied to predict the refined feature map.
Revised: Similarly, another convolutional layer with 384 input channels and 256 output channels is applied to predict the refined feature map.

Original: (g) Upsampling 8->4.
Revised: (g) Upsampling 8->4.

Original: We upsample the feature map from (f) to stride 4.
Revised: We upsample the feature map from (f) to stride 4.

Original: The procedure is the same as (e) except for the number of channels.
Revised: The procedure is the same as (e) except for the number of channels.

Original: More specifically, we first upsample the feature map and concatenate with the feature map from the encoder with consistent stride, obtaining a feature map with 256 + 192 = 448 channels and stride 4.
Revised: More specifically, we first upsample the feature map and concatenate with the feature map from the encoder with consistent stride, obtaining a feature map with 256 + 192 = 448 channels and stride 4.

Original: Then a convolutional layer with 448 input channels, 448 output channels, and 4 groups is applied to fuse the information.
Revised: Then a convolutional layer with 448 input channels, 448 output channels, and 4 groups is applied to fuse the information.

Original: Then the result feature map is transformed by the instance normalization and LeakyReLU activation function.
Revised: Then the result feature map is transformed by the instance normalization and LeakyReLU activation function.

Original: Next, we apply another convolutional layer with 448 input channels and 64 output channels to compress the feature channels.
Revised: Next, we apply another convolutional layer with 448 input channels and 64 output channels to compress the feature channels.

Original: Again, the instance normalization layer and LeakyReLU function is applied.
Revised: Again, the instance normalization layer and LeakyReLU function is applied.

Original: In the end, we add a skip connection between the final feature map and the previous concatenated feature map.
Revised: In the end, we add a skip connection between the final feature map and the previous concatenated feature map.

Original: (h) Refine 4.
Revised: (h) Refine 4.

Original: We refine the depth map from (f) and the feature map from (g) following the same pipeline as (d) .
Revised: We refine the depth map from (f) and the feature map from (g) following the same pipeline as (d) .

Original: We first upsample the depth map to stride 4, and concatenate with the feature map, obtaining a new feature map with 64 + 128 = 192 channels and stride 4.
Revised: We first upsample the depth map to stride 4, and concatenate with the feature map, obtaining a new feature map with 64 + 128 = 192 channels and stride 4.

Original: Next, we apply a convolutional layer with 192 input channels and 192 output channels to fuse the information, and a LeakyReLU function to transform the feature map.
Revised: Next, we apply a convolutional layer with 192 input channels and 192 output channels to fuse the information, and a LeakyReLU function to transform the feature map.

Original: In the end, we utilize another convolutional layer with 192 input
Revised: In the end, we utilize another convolutional layer with 192 input

Original: channels
Revised: channels

Original: and 128 output channels to predict the new depth map.
Revised: and 128 output channels to predict the new depth map.

Original: The output is a
Revised: The output is a

Original: depth map
Revised: depth map

Original: with
Revised: with

Original: 128 channels
Revised: 128 channels

Original: and
Revised: and

Original: stride 4.
Revised: stride 4.

Original: (i) Metric Layer.
Revised: (i) Metric Layer.

Original: We take the feature map with stride 32 from the encoder as input.
Revised: We take the feature map with stride 32 from the encoder as input.

Original: We first apply global max pooling to compress the feature map into a vector with 1536 channels.
Revised: We first apply global max pooling to compress the feature map into a vector with 1536 channels.

Original: Then we apply a fully-connected layer with 1536 input units and 384 output units to compress the channels.
Revised: Then we apply a fully-connected layer with 1536 input units and 384 output units to compress the channels.

Original: Next we apply a LeakyReLU function.
Revised: Next we apply a LeakyReLU function.

Original: In the end, we utilize another fully-connected layer with 384 input units and 2 output units to regress the scale and shift.
Revised: In the end, we utilize another fully-connected layer with 384 input units and 2 output units to regress the scale and shift.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: (j) Final Prediction.
Revised: (j) Final Prediction.

Original: We take the output from (d) , (f) , (h) and (i) as input.
Revised: We take the output from (d) , (f) , (h) and (i) as input.

Original: We upsample all the depth maps to stride 1 via bi-linear interpolation, and fuse the depth maps into a single depth map by the addition operation and a convolutional layer with 128 input channels and 1 output channel.
Revised: We upsample all the depth maps to stride 1 via bi-linear interpolation, and fuse the depth maps into a single depth map by the addition operation and a convolutional layer with 128 input channels and 1 output channel.

Original: In the end, we add the depth map with the shift and multiply with the scale from (i) , respectively.
Revised: In the end, we add the depth map with the shift and multiply with the scale from (i) , respectively.

Original: A. S TATISTICAL E VALUATION W ITH O
Revised: A. S TATISTICAL E VALUATION W ITH O

Original: THER
Revised: THER

Original: M ETHODS
Revised: M ETHODS

Original: In this section, we compare with Ramamonjisoa et al.
Revised: In this section, we compare with Ramamonjisoa et al.

Original: (2020), Cheng et al.
Revised: (2020), Cheng et al.

Original: (2018), and Li et al.
Revised: (2018), and Li et al.

Original: (2017) on NYU Depth V2 (Silberman et al., 2012).
Revised: (2017) on NYU Depth V2 (Silberman et al., 2012).

Original: More specifically, we apply their methods to refine the final predicted depth map.
Revised: More specifically, we apply their methods to refine the final predicted depth map.

Original: The displacement field, the affinity matrix, and the depth gradient are predicted from the refined feature map, which is the output of the last refinement module.
Revised: The displacement field, the affinity matrix, and the depth gradient are predicted from the refined feature map, which is the output of the last refinement module.

Original: For
Revised: For

Original: Cheng et al.
Revised: Cheng et al.

Original: we adopt their default hyper-parameters.
Revised: we adopt their default hyper-parameters.

Original: The kernel size is 3, and the number of iterations is 24.
Revised: The kernel size is 3, and the number of iterations is 24.

Original: Li et al.
Revised: Li et al.

Original: (2017) proposes the end-to-end fusion strategy and the optimization-based fusion strategy.
Revised: (2017) proposes the end-to-end fusion strategy and the optimization-based fusion strategy.

Original: For the end-to-end fusion strategy, we employ three convolutional layers with kernel size 5, and 16 channels.
Revised: For the end-to-end fusion strategy, we employ three convolutional layers with kernel size 5, and 16 channels.

Original: For the optimization-based fusion strategy, we use Adam optimizer.
Revised: For the optimization-based fusion strategy, we use Adam optimizer.

Original: The learning rate is 0 .
Revised: The learning rate is 0 .

Original: and the number of iterations is 100.
Revised: and the number of iterations is 100.

Original: The results are shown in Tab.
Revised: The results are shown in Tab.

Original: Our method achieves better performance.
Revised: Our method achieves better performance.

Original: A. A BLATION ON R ESOLUTION
Revised: A. A BLATION ON R ESOLUTION

Original: In this part, we evaluate the performance of the V-layer when applying on the feature map with stride 8.
Revised: In this part, we evaluate the performance of the V-layer when applying on the feature map with stride 8.

Original: We achieve this by dividing the feature map into 4 rectangle sub-regions with equal area (top left, top right, bottom left and bottom right).
Revised: We achieve this by dividing the feature map into 4 rectangle sub-regions with equal area (top left, top right, bottom left and bottom right).

Original: We apply a V-layer on each sub-region.
Revised: We apply a V-layer on each sub-region.

Original: The predicted depth maps from the sub-regions will be stacked to recover the original resolution.
Revised: The predicted depth maps from the sub-regions will be stacked to recover the original resolution.

Original: The results are shown in Tab.(10).
Revised: The results are shown in Tab.(10).

Original: From stride 16 to 8, the improvement of accuracy is marginal.
Revised: From stride 16 to 8, the improvement of accuracy is marginal.

Original: However, the inference speed (FPS) is significantly slow down.
Revised: However, the inference speed (FPS) is significantly slow down.

Original: A. A BLATION ON KITTI AND SUN RGB-D
Revised: A. A BLATION ON KITTI AND SUN RGB-D

Original: To further demonstrate the benefits of our V-layer on KITTI Eigen split (Geiger et al., 2012; Eigen et al., 2014) and SUN RGB-D test set (Song et al., 2015), we present the accuracy of the predicted depth map when replacing the V-layer with a single convolutional layer.
Revised: To further demonstrate the benefits of our V-layer on KITTI Eigen split (Geiger et al., 2012; Eigen et al., 2014) and SUN RGB-D test set (Song et al., 2015), we present the accuracy of the predicted depth map when replacing the V-layer with a single convolutional layer.

Original: As shown in Tab.(11), our
Revised: As shown in Tab.(11), our

Original: V-layer can improve the depth map accuracy on both KITTI and SUN
Revised: V-layer can improve the depth map accuracy on both KITTI and SUN

Original: RGB-D.
Revised: RGB-D.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: A. M ORE G ENERALIZATION E XPERIMENTS
Revised: A. M ORE G ENERALIZATION E XPERIMENTS

Original: We further evaluated the generalization performance when training on KITTI and test on NYU, or training on NYU and test on KITTI.
Revised: We further evaluated the generalization performance when training on KITTI and test on NYU, or training on NYU and test on KITTI.

Original: The results are shown in Tab.(12) and Tab.(13).
Revised: The results are shown in Tab.(12) and Tab.(13).

Original: Our method achieves better generalization performance in both settings.
Revised: Our method achieves better generalization performance in both settings.

Original: [Table]
Revised: [Table]

Original: A. S PARSE G ROUND -T RUTH D EPTH M AP
Revised: A. S PARSE G ROUND -T RUTH D EPTH M AP

Original: KITTI provides sparse LiDAR measurements as the ground truth (official ground truth has been inpainted to an extent).
Revised: KITTI provides sparse LiDAR measurements as the ground truth (official ground truth has been inpainted to an extent).

Original: Yet, our method could work well in such cases where sparse LiDAR measurements are known.
Revised: Yet, our method could work well in such cases where sparse LiDAR measurements are known.

Original: Our algorithm computes the depth map from the gradient predicted by the network in a differentiable way.
Revised: Our algorithm computes the depth map from the gradient predicted by the network in a differentiable way.

Original: Thereby, for such a case, when we apply sparse supervision on the computed depth map (note that the loss function is used for valid measurements only), the error signal from the loss function will be back-propagated to the predicted gradient to supervise the network
Revised: Thereby, for such a case, when we apply sparse supervision on the computed depth map (note that the loss function is used for valid measurements only), the error signal from the loss function will be back-propagated to the predicted gradient to supervise the network

Original: [Figure]
Revised: [Figure]

Original: to learn gradient clues.
Revised: to learn gradient clues.

Original: We demonstrated the effectiveness of our algorithm in Tab.(2) and Tab.(3) for such cases.
Revised: We demonstrated the effectiveness of our algorithm in Tab.(2) and Tab.(3) for such cases.

Original: A. M ORE V ISUALIZATION
Revised: A. M ORE V ISUALIZATION

Original: We visualize the confidence weight map Σ x , the difference map Γ x , and the depth map Z u from the
Revised: We visualize the confidence weight map Σ x , the difference map Γ x , and the depth map Z u from the

Original: V-layer in Fig.5.
Revised: V-layer in Fig.5.

Original: We observe that the depth value of a pixel shows correlation with respect to the image coordinates of the pixel.
Revised: We observe that the depth value of a pixel shows correlation with respect to the image coordinates of the pixel.

Original: For example, in the last example in Fig.5, for different pixels at the door, the depth values are usually different but the first order difference are approximately the same.
Revised: For example, in the last example in Fig.5, for different pixels at the door, the depth values are usually different but the first-order difference are approximately the same.

Original: This observation shows that the difference map might be easier to predict than the depth map.
Revised: This observation shows that the difference map might be easier to predict than the depth map.

Original: A. Q UALITATIVE R ESULTS
Revised: A. Q UALITATIVE R ESULTS

Original: We provide more qualitative results on KITTI Eigen split (Eigen et al., 2014), SUN RGB-D (Song et al., 2015), and NYU Depth V2 (Silberman et al., 2012) in Fig.6, Fig.7 and Fig.8, respectively.
Revised: We provide more qualitative results on KITTI Eigen split (Eigen et al., 2014), SUN RGB-D (Song et al., 2015), and NYU Depth V2 (Silberman et al., 2012) in Fig.6, Fig.7 and Fig.8, respectively.

Original: Our framework predicts more accurate shapes and preserves the high-frequency scene information.
Revised: Our framework predicts more accurate shapes and preserves the high-frequency scene information.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]