Original: SurroundDepth: Entangling Surrounding Views for Self-Supervised Multi-Camera Depth Estimation
Revised: SurroundDepth: Entangling Surrounding Views for Self-Supervised Multi-Camera Depth Estimation

Original: Anonymous Author(s)

Original: AffiliationAddress email Abstract: Depth estimation from images serves as the fundamental step of 3Dperception for autonomous driving and is an economical alternative to expensive depth sensors like LiDAR.
Revised: Abstract: Depth estimation from images serves as the fundamental step of 3D perception for autonomous driving and is an economical alternative to expensive depth sensors like LiDAR.

Original: The temporal photometric constraints enablesself-supervised depth estimation without labels, further facilitating its application.
Revised: The temporal photometric constraints enables self-supervised depth estimation without labels, further facilitating its application.

Original: However, most existing methods predict the depth solely based on each monocularimage and ignore the correlations among multiple surrounding cameras, which aretypically available for modern self-driving vehicles.
Revised: However, most existing methods predict the depth solely based on each monocular image and ignore the correlations among multiple surrounding cameras, which are typically available for modern self-driving vehicles.

Original: In this paper, we propose a
Revised: In this paper, we propose a

Original: SurroundDepth method to incorporate the information from multiple surroundingviews to predict depth maps across cameras.
Revised: SurroundDepth method to incorporate the information from multiple surrounding views to predict depth maps across cameras.

Original: Specifically, we employ a joint network to process all the surrounding views and propose a cross-view transformerto effectively fuse the information from multiple views.
Revised: Speciﬁcally, we employ a joint network to process all the surrounding views and propose a cross-view transformer to effectively fuse the information from multiple views.

Original: We apply cross-view selfattention to efficiently enable the global interactions between multi-camera featuremaps.
Revised: We apply cross-view selfattention to efﬁciently enable the global interactions between multi-camera feature maps.

Original: Different from self-supervised monocular depth estimation, we are able topredict real-world scales given multi-camera extrinsic matrices.
Revised: Different from self-supervised monocular depth estimation, we are able to predict real-world scales given multi-camera extrinsic matrices.

Original: To achieve thisgoal, we adopt the two-frame structure-from-motion to extract scale-aware pseudodepths to pretrain the models.
Revised: To achieve this goal, we adopt the two-frame structure-from-motion to extract scale-aware pseudo depths to pretrain the models.

Original: Further, instead of predicting the ego-motion ofeach individual camera, we estimate a universal ego-motion of the vehicle andtransfer it to each view to achieve multi-view ego-motion consistency.
Revised: Further, instead of predicting the ego-motion of each individual camera, we estimate a universal ego-motion of the vehicle and transfer it to each view to achieve multi-view ego-motion consistency.

Original: In experiments, our method achieves the state-of-the-art performance on the challengingmulti-camera depth estimation datasets DDAD and nuScenes.
Revised: In experiments, our method achieves the state-of-the-art performance on the challenging multi-camera depth estimation datasets DDAD and nuScenes.

Revised: Code is available at https://github.com/weiyithu/SurroundDepth.

Original:  Introduction
Revised:  Introduction

Original: Recent years have witnessed the rapid development of autonomous driving.
Revised: Recent years have witnessed the rapid development of autonomous driving.

Original: Instead of relying onexpensive depth sensors like LiDAR to perform extract structural information, 3D perception fromcameras has become a promising approach and potential alternative due to its semantic richnessand economy.
Revised: Instead of relying on expensive depth sensors like LiDAR to perform extract structural information, 3D perception from cameras has become a promising approach and potential alternative due to its semantic richness and economy.

Original: Acting as a bridge between the input 2D image and the real 3D environment, depthestimation has a crucial influence on the downstream 3D understanding and receives increasingattention [1, 2, 3, 4, 5, 6, 7, 8, 9, 10].
Revised: Acting as a bridge between the input 2D image and the real 3D environment, depth estimation has a crucial inﬂuence on the downstream 3D understanding and receives increasing attention [1, 2, 3, 4, 5, 6, 7, 8, 9, 10].

Original: Due to the expensive cost of densely annotated depth maps, depth estimation is usually learned in aself-supervised manner.
Revised: Due to the expensive cost of densely annotated depth maps, depth estimation is usually learned in a self-supervised manner.

Original: By simultaneously predicting the depth maps and the ego-motions of cameras, existing methods take advantage of the temporal photometric constraints between successiveimages as the supervision signal [11, 12, 13, 14, 15, 16].
Revised: By simultaneously predicting the depth maps and the ego-motions of cameras, existing methods take advantage of the temporal photometric constraints between successive images as the supervision signal [11, 12, 13, 14, 15, 16].

Original: Despite that modern self-driving carsare usually equipped with multiple cameras to capture the full 360 ◦ view of the surrounding scene,most existing methods still focus on predicting depth maps from monocular images and ignore thecorrelations [17, 18, 19, 20, 21, 22, 23, 24, 25, 26] among the surrounding views [27, 28, 29, 30, 31].
Revised: Despite that modern self-driving cars

Original: S e l f - S u p e r v
Revised: S e l f - S u p e r v

Original: i s e d
Revised: i s e d

Original: D e p t h
Revised: D e p t h

Original: E s t i m a t i o n Conventional: Individual methods. Conventional methods [27, 28] predict depths and ego-motions of each view separately, which ignores the correlations between views. Our method incorporates theinformation across cameras and jointly process all surrounding views.
Revised: E s t i m a t i o n are usually equipped with multiple cameras to capture the full 360 ◦ view of the surrounding scene, most existing methods still focus on predicting depth maps from monocular images and ignore the correlations [17, 18, 19, 20, 21, 22, 23, 24, 25, 26] among the surrounding views [27, 28, 29, 30, 31].

Original: In this paper, we propose SurroundDepth to process all the surrounding views jointly to producehigh-quality depth maps across cameras.
Revised: In this paper, we propose SurroundDepth to process all the surrounding views jointly to produce high-quality depth maps across cameras.

Original: We first employ a shared encoder to extract high-levelfeature maps for each view and then propose a cross-view transformer to effectively fuse them.
Revised: We ﬁrst employ a shared encoder to extract high-level feature maps for each view and then propose a cross-view transformer to effectively fuse them.

Original: Toefficiently allow multi-scale interactions between the features across cameras, we first reduce theresolution of feature maps with depthwise separable convolutions.
Revised: To efﬁciently allow multi-scale interactions between the features across cameras, we ﬁrst reduce the resolution of feature maps with depthwise separable convolutions.

Original: Then we apply cross-view attention to integrate multi-camera features and use deconvolution to recover the original resolution.
Revised: Then we apply cross-view attention to integrate multi-camera features and use deconvolution to recover the original resolution.

Original: To alleviate the information loss induced by feature map downsampling, we add skip connectionsbetween individual and interacted feature maps.
Revised: To alleviate the information loss induced by feature map downsampling, we add skip connections between individual and interacted feature maps.

Original: Finally, we use a shared decoder to attain thepredicted depth maps.
Revised: Finally, we use a shared decoder to attain the predicted depth maps.

Original: To recover the real-world scales, we propose Structure-from-Motion (SfM)pretraining and joint pose estimation to predict scale-aware results.
Revised: To recover the real-world scales, we propose Structure-from-Motion (SfM) pretraining and joint pose estimation to predict scale-aware results.

Original: In detail, we pretrain the depthestimation model with scale-aware pseudo depths derived from SfM [32].
Revised: In detail, we pretrain the depth estimation model with scale-aware pseudo depths derived from SfM [32].

Original: Joint pose estimation isdesigned to achieve consistent pose predictions between cameras.
Revised: Joint pose estimation is designed to achieve consistent pose predictions between cameras.

Original: By explicitly leveraging extrinsicmatrices, we are able to calculate the individual ego-motion of each view from the predicted universal ego-motion.
Revised: By explicitly leveraging extrinsic matrices, we are able to calculate the individual ego-motion of each view from the predicted universal ego-motion.

Original: Extensive experiments on the challenging multi-camera depth estimation datasetdemonstrate the effectiveness of the proposed method.
Revised: We conduct extensive experiments on the challenging multi-camera depth estimation datasets DDAD [27] and nuScenes [33]. Experimental results show that the cross-view feature interaction booost the performance and the proposed techniques can achieve real-scale depth estimation.

Original:  Related Work
Revised:  Related Work

Original: Self-supervised Monocular Depth Estimation: Without available ground-truths, many approachesexplore the routes of learning depths and motions simultaneously [11, 28, 12, 13, 14, 33, 15, 16, 34].
Revised: Self-supervised Monocular Depth Estimation: Without available ground-truths, many approaches explore the routes of learning depths and motions simultaneously [11, 28, 12, 13, 14, 34, 15, 16, 35].

Original: For monocular sequences, the geometric constraints are usually built on adjacent frames.
Revised: For monocular sequences, the geometric constraints are usually built on adjacent frames.

Original: Zhou etal.
Revised: Zhou et al.

Original: [11] built the problem as a task of view synthesis and trained two networks to separately predict poses and depths.
Revised: [11] built the problem as a task of view synthesis and trained two networks to separately predict poses and depths.

Original: Monodepth

Revised: Monodepth

Original: [28] further enhanced the quality of predictions by proposingthe minimum re-projection loss, full-resolution multi-scale sampling, and auto-masking loss, whichhas been adopted in [35, 36].
Revised: [28] further enhanced the quality of predictions by proposing the minimum re-projection loss, full-resolution multi-scale sampling, and auto-masking loss, which has been adopted in [36, 37].

Original: Recently, FSM [29] extended self-supervised monocular depth estimation to full surrounding views by introducing both spatial and temporal contexts to enrich thesupervision signals.
Revised: Recently, FSM [29] extended self-supervised monocular depth estimation to full surrounding views by introducing both spatial and temporal contexts to enrich the supervision signals.

Original: Different from these monocular depth estimation methods, Our SurroundDepthcaptures cross-view interactions in surrounding views, which are important for fully understandingthe environment with multiple cameras.
Revised: Different from these monocular depth estimation methods, Our SurroundDepth captures cross-view interactions in surrounding views, which are important for fully understanding the environment with multiple cameras.

Original: Additional Supervision for Depth Estimation:
Revised: Additional Supervision for Depth Estimation:

Original: Recent approaches introduce additional supervision signals to strengthen the accuracy of depth estimation, such as optical flow [14, 37] and objectmotion [35, 16].
Revised: Recent approaches introduce additional supervision signals to strengthen the accuracy of depth estimation, such as optical ﬂow [14, 38] and object motion [36, 16].

Original: DispNet [38] was the first work to transfer information from synthetic stereo videodatasets to the real-word depth estimation.
Revised: DispNet [39] was the ﬁrst work to transfer information from synthetic stereo video datasets to the real-word depth estimation.

Original: Besides, Zheng et al.
Revised: Besides, Zheng et al.

Original: [39] proposed a two-module domain adaptive network with a generative adversarial loss to transfer knowledge from the syntheticdomain.
Revised: [40] proposed a two-module domain adaptive network with a generative adversarial loss to transfer knowledge from the synthetic domain.

Original: Some methods adopt auxiliary depth sensors to capture accurate depths, such as LiDAR,to assist depth estimation [40, 41].
Revised: Some methods adopt auxiliary depth sensors to capture accurate depths, such as LiDAR, to assist depth estimation [41, 42].

Original: To predict the depths with real-world scales, our method leverages multi-camera extrinsic matrices and takes the sparse depths obtained from SfM to pretrain ournetwork.
Revised: To predict the depths with real-world scales, our method lever-

Original: [Figure]
Revised: [Figure]

Revised: ages multi-camera extrinsic matrices and takes the sparse depths obtained from SfM to pretrain our network.

Original:  Approach 
Revised:  Approach 3.1 Problem Formulation

Original: 3.1 Problem Formulation

Original: In the self-supervised depth and ego-motion settings, the depth network F and pose network G areoptimized by minimizing a per-pixel photometric reprojection loss [42, 11].
Revised: In the self-supervised depth and ego-motion settings, the depth network F and pose network G are optimized by minimizing a per-pixel photometric reprojection loss [43, 11].

Original: This reconstructionbased self-supervision paradigm achieves great progress in monocular depth estimation methods [11,12]
Revised: This reconstructionbased self-supervision paradigm achieves great progress in monocular depth estimation methods [11, 12]

Original: [Figure]
Revised: [Figure]

Original: and can be directly extended into multi-camera full surround depth estimation.
Revised: and can be directly extended into multi-camera full surround depth estimation.

Original: The predicteddepth maps and poses of a set of input surrounding samples I =
Revised: The predicted depth maps and poses of a set of input surrounding samples I =

Original: { I 1 , I 2 , · · · I N } can be written as:
Revised: { I 1 , I 2 , · · · I N } can be written as:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where subscript s and t mean the source and target frames.
Revised: where subscript s and t mean the source and target frames.

Original: However, estimating full surroundingdepths differs from monocular depth estimation in that there exists crucial correlations among thesurrounding views, as shown in Figure 1.
Revised: However, estimating full surrounding depths differs from monocular depth estimation in that there exists crucial correlations among the surrounding views, as shown in Figure 1.

Original: The overlaps between adjacent views connect all viewsinto a full 360 ◦ view of the surroundings, which contains lots of beneficial knowledge and priors forunderstanding the whole scene.
Revised: The overlaps between adjacent views connect all views into a full 360 ◦ view of the surroundings, which contains lots of beneﬁcial knowledge and priors for understanding the whole scene.

Original: Based on this fact, we build the joint models to predict depths andego-motions instead of view-dependent estimations:
Revised: Based on this fact, we build the joint models to predict depths and ego-motions instead of view-dependent estimations:

Original: [Equation]
Revised: [Equation]

Original: Profiting from the joint model, we can not only enable the cross-view information interactions toimprove the performance of all views but also generate the universal ego-motion to produce scaleaware predictions with camera extrinsic matrices.
Revised: Proﬁting from the joint model, we can not only enable the cross-view information interactions to improve the performance of all views but also generate the universal ego-motion to produce scaleaware predictions with camera extrinsic matrices.

Original: 3.2 Overview
Revised: 3.2 Overview

Original: As illustrated in Figure 2, the network F can be separated into three parts ( i.e. , a shared encoder
Revised: As illustrated in Figure 2, the network F can be separated into three parts ( i.e. , a shared encoder

Original: E , a shared decoder D , and several cross-view transformers (CVT)).
Revised: E , a shared decoder D , and several cross-view transformers (CVT)).

Original: Given a set of surroundingimages, the encoder networks first extract their multi-scale representations in parallel.
Revised: Given a set of surrounding images, the encoder networks ﬁrst extract their multi-scale representations in parallel.

Original: Different fromexisting methods that directly decode the learned features, we entangle the features from all viewsinto an integrated feature at each scale, and further utilize multiple scale-specific CVT to performcross-view self-attentions over all scales.
Revised: Different from existing methods that directly decode the learned features, we entangle the features from all views into an integrated feature at each scale, and further utilize multiple scale-speciﬁc CVT to perform cross-view self-attentions over all scales.

Original: By taking advantage of the powerful attention mechanism,
Revised: By taking advantage of the powerful attention mechanism,

Original: CVT enables each element on the feature maps to propagate its information to other positions andabsorb information from others at the same time.
Revised: CVT enables each element on the feature maps to propagate its information to other positions and absorb information from others at the same time.

Original: Finally, we separate the post-interactive featuresback to N views and send them to the decoder D .
Revised: Finally, we separate the post-interactive features back to N views and send them to the decoder D .

Original: Unlike monocular depth estimation, we are able to recover the real-world scale from camera extrinsicmatrices.
Revised: Unlike monocular depth estimation, we are able to recover the real-world scale from camera extrinsic matrices.

Original: A straightforward method to leverage these camera extrinsic matrices is to embed theminto spatial photometric loss between two adjacent views.
Revised: A straightforward method to leverage these camera extrinsic matrices is to embed them into spatial photometric loss between two adjacent views.

Original: However, we find that the depth networkfails to learn the scale directly supervised by spatial photometric loss.
Revised: However, we ﬁnd that the depth network fails to learn the scale directly supervised by spatial photometric loss.

Original: To tackle this issue, wepropose scale-aware SfM pretraining and joint pose estimation.
Revised: To tackle this issue, we propose scale-aware SfM pretraining and joint pose estimation.

Original: Specifically, we use two-frame SfMto generate pseudo depths to pretrain the models.
Revised: Speciﬁcally, we use two-frame SfM to generate pseudo depths to pretrain the models.

Original: The pretrained depth network is able to learnthe real-world scale.
Revised: The pretrained depth network is able to learn the real-world scale.

Original: Moreover, the temporal ego-motions of N cameras have explicit geometricconstraints.
Revised: Moreover, the temporal ego-motions of N cameras have explicit geometric constraints.

Original: Instead of using a consistency loss, we estimate the universal pose of the vehicle andcalculate the ego-motion of each view according to their extrinsic matrices.
Revised: Instead of using a consistency loss, we estimate the universal pose of the vehicle and calculate the ego-motion of each view according to their extrinsic matrices.

Original: 3.3 Cross-View Transformer
Revised: 3.3 Cross-View Transformer

Original: With the multi-scale features extracted from all surrounding views, we replace the skip connectionsbetween the encoder and decoder with our proposed cross-view transformer (CVT), as shown in
Revised: With the multi-scale features extracted from all surrounding views, we replace the skip connections between the encoder and decoder with our proposed cross-view transformer (CVT), as shown in

Original: Figure 3.
Revised: Figure 3.

Original: Let X k ∈ R N ×
Revised: Let X k ∈ R N ×

Original: H k × W k × d k be the feature maps obtained from the k -th scale, and wefirst supplement each feature with three unique learnable positional encodings ( PE N , PE H , PE W ),which represent the view-wise, row-wise and column-wise index, respectively.
Revised: H k × W k × d k be the feature maps obtained from the k -th scale, and we ﬁrst supplement each feature with three unique learnable positional encodings ( PE N , PE H , PE W ), which represent the view-wise, row-wise and column-wise index, respectively.

Original: Then we build Z cross-view self-attention layers to perform the cross-view information exchanging,whose computation cost is O
Revised: Then we build Z cross-view self-attention layers to perform the cross-view information exchanging, whose computation cost is O

Original: ( H 2 k W 2 k d k ) for each layer.
Revised: ( H 2 k W 2 k d k ) for each layer.

Original: However, the cost would be too enormous toaccept when processing larger feature maps from the lower layers of the encoder.
Revised: However, the cost would be too enormous to accept when processing larger feature maps from the lower layers of the encoder.

Original: To avoid the hugecost caused by the quadratic term ( i.e. , H 2 k W 2 k ), we place a depthwise separable convolution (DSConv) [43, 44] before attention layers to first summarize the large feature maps into lower-resolutionones with same channel numbers, i.e. , X ′ k ∈ R N × h k × w k × d k .
Revised: To avoid the huge cost caused by the quadratic term ( i.e. , H 2 k W 2 k ), we place a depthwise separable convolution (DSConv) [44, 45] before attention layers to ﬁrst summarize the large feature maps into lower-resolution ones with same channel numbers, i.e. , X (cid:48) k ∈ R N × h k × w k × d k .

Original: With much less computation than astandard convolution, the DS-Conv is widely adopted to balance the trade-off between efficiency andperformance.
Revised: With much less computation than a standard convolution, the DS-Conv is widely adopted to balance the trade-off between efﬁciency and performance.

Original: To create the pathway across the features of surrounding views, we flatten the featuremaps into an unified sequence, which includes the elements from all views, i.e. , N × h k × w kelements in total.
Revised: To create the pathway across the features of surrounding views, we ﬂatten the feature maps into an uniﬁed sequence, which includes the elements from all views, i.e. , N × h k × w k elements in total.

Original: We develop three linear layers to obtain the query, key, and value vectors from
Revised: We develop three linear layers to obtain the query, key, and value vectors from

Original: X ′ k
Revised: X (cid:48) k

Original: + PE N + PE H
Revised: + PE N + PE H

Original: + PE W ,
Revised: + PE W ,

Original: and then we split the features into multiple groups along the channeldimension for multi-head self-attention to enrich the feature diversity.
Revised: and then we split the features into multiple groups along the channel dimension for multi-head self-attention to enrich the feature diversity.

Original: The output features can berepresented as:
Revised: The output features can be represented as:

Original: [Equation]
Revised: [Equation]

Original:  where M is the number of feature groups, and 
Revised: (3) where M is the number of feature groups, and K i , Q i ,

Original: K i , Q i , V i denote the i -th feature group of the key,query, and value features, respectively.
Revised: V i denote the i -th feature group of the key, query, and value features, respectively.

Original: Benefiting from this attention unit, we ensure that everyelement on X ′ k not only interacts with intra-view features for better understanding of the currentframe but also receives inter-view contexts for comprehensive cognition of the scene.
Revised: Beneﬁting from this attention unit, we ensure that every element on X (cid:48) k not only interacts with intra-view features for better understanding of the current frame but also receives inter-view contexts for comprehensive cognition of the scene.

Original: To makecross-view information fully interactive, we stack such unit for Z times in a cascade style. Asan inverse process of flattening, we reshape the outputs of the Z -th attention layers back to the
Revised: To make

Original: D S C on v ...
Revised: D S C on v ...

Original: Cross-View Self-Attention × Z ...
Revised: Cross-View Self-Attention × Z ...

Original: D S D e C on v ...
Revised: D S D e C on v ...

Original: Skip Connection
Revised: Skip Connection

Original: N×H k
Revised: N×H k

Original: ×W k ×d k
Revised: ×W k ×d k

Original: N×h k ×w k ×d k
Revised: N×h k ×w k ×d k

Original: N×H k
Revised: N×H k

Original: ×W k ×d k
Revised: ×W k ×d k

Original: N×h k ×w k ×d kshape of N × h k × w k × d k , and further upsample the features to the original resolution of inputfeatures through a depthwise separable deconvolution. To alleviate the information loss caused bydownsampling the feature maps and preserve the input details, we construct a skip connection todirectly combine the input and output features:
Revised: N×h k ×w k ×d k

Original: [Equation]
Revised: [Equation]

Original: By extending this strategy, features of all scales and allviews can be fully propagated and updatedfrom a global perspective.
Revised: By extending this strategy features of all scales and all can be fully propagated and updated from a global perspective.

Original: 3.4 Scale-aware Structure-from-Motion Pretraining
Revised: 3.4 Scale-aware Structure-from-Motion Pretraining

Original: The aim of SfM pretraining is to explore the real-world scale from camera extrinsic matrices.
Revised: The aim of SfM pretraining is to explore the real-world scale from camera extrinsic matrices.

Original: Adirect way to leverage extrinsic matrices is to use spatial photometric loss between two neighboringviews, i.e. , warping I it to I jt :
Revised: A direct way to leverage extrinsic matrices is to use spatial photometric loss between two neighboring views, i e. , warping I it to I jt :

Original: [Equation]
Revised: [Equation]

Original: (5) where K i, T i are the intrinsic and extrinsic matrices of i th camera.
Revised: [Equation] where K i T i are the intrinsic and extrinsic matrices of i th camera.

Original: However, the overlap between
Revised: However, the overlap between

Original: I i and I j is relatively small, and the p i → j t is easy to be out of the image bounds if the scale of D itis far from the real-world scale.
Revised: I i and I j is relatively small, and the p i → j t is easy to be out of the image bounds if the scale of D it is far from the real-world scale.

Original: In this way, at the start of the training, the spatial photometric losswill be invalid and cannot supervise the depth network to learn the scale.
Revised: In this way, at the start of the training, the spatial photometric loss will be invalid and cannot supervise the depth network to learn the scale.

Original: To tackle the problem, we first adopt SIFT [45] descriptors to extract correspondences.
Revised: To tackle the problem, we ﬁrst adopt SIFT [46] descriptors to extract correspondences.

Original: Then wecompute scale-aware pseudo depths by triangulation with camera extrinsic matrices.
Revised: Then we compute scale-aware pseudo depths by triangulation with camera extrinsic matrices.

Original: Finally, weleverage these sparse pseudo depths along with the temporal photometric loss to pretrain the depthand pose networks.
Revised: Finally, we leverage these sparse pseudo depths along with the temporal photometric loss to pretrain the depth and pose networks.

Original: However, due to the small overlap and large view changes, the robustnessand accuracy of descriptors decrease.
Revised: However, due to the small overlap and large view changes, the robustness and accuracy of descriptors decrease.

Original: To alleviate the issue, we only find corresponding points ina certain region instead of the whole image since we can roughly know the overlapping regionsbetween two neighboring views.
Revised: To alleviate the issue, we only ﬁnd corresponding points in a certain region instead of the whole image since we can roughly know the overlapping regions between two neighboring views.

Original: Specifically, in this work, we define left 13 part of I i and right13 part of I i +1 as the valid regions.
Revised: Speciﬁcally, in this work, we deﬁne left 13 part of I i and right 13 part of I i +1 as the valid regions.

Original: Further, we leverage epipolar geometry to filter outliers.
Revised: Further, we leverage epipolar geometry to ﬁlter outliers.

Original: Theepipolar line of point q im can be described as:
Revised: The epipolar line of point q im can be described as:

Original: [Equation]
Revised: [Equation]

Original: 6) where F i → j is the fundamental matrix.
Revised: where F i → j is the fundamental matrix.

Original: If the distance between q jm and epipolar line l m is largerthan a threshold γ , we regard this correspondence pair as the noise.
Revised: If the distance between q jm and epipolar line l m is larger than a threshold γ , we regard this correspondence pair as the noise.

Original: We provide an example in thesupplementary materials.
Revised: We provide an example in the supplementary materials.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 3.5 Joint Pose Estimation
Revised: 3.5 Joint Pose Estimation

Original: An intuitive way of pose estimation is to predict the relative pose of each view separately, whichcan be represented by Equation 1.
Revised: An intuitive way of pose estimation is to predict the relative pose of each view separately, which can be represented by Equation 1.

Original: However, this strategy ignores the pose consistency betweendifferent views, which may lead to ineffective supervision signals.
Revised: However, this strategy ignores the pose consistency between different views, which may lead to ineffective supervision signals.

Original: To maintain multi-view egomotion consistency, we propose to decompose the camera pose estimation problem into two subproblems: universal pose prediction and universal-to-local transformation.
Revised: To maintain multi-view egomotion consistency, we propose to decompose the camera pose estimation problem into two subproblems: universal pose prediction and universal-to-local transformation.

Original: To obtain the universalpose P , we feed N pairs of target and source images into the PoseNet G at a time and take theaverage of extracted features before the decoder.
Revised: To obtain the universal pose P , we feed N pairs of target and source images into the PoseNet G at a time and take the average of extracted features before the decoder.

Original: The universal pose can be computed by:
Revised: The universal pose can be computed by:

Original: [Equation]
Revised: [Equation]

Original: where G E and G D denote the encoder and decoder of PoseNet, respectively.
Revised: where G E and G D denote the encoder and decoder of PoseNet, respectively.

Original: After obtaining theuniversal pose P , we can further transform it to each camera pose with known camera extrinsicmatrices, which can be described as:
Revised: After obtaining the universal pose P , we can further transform it to each camera pose with known camera extrinsic matrices, which can be described as:

Original: [Equation]
Revised: [Equation]

Original: 8) where T i is the extrinsic matrix of i th camera and P it → s is its ego-motion from target frame to sourceframe.
Revised: (8) where T i is the extrinsic matrix of i th camera and P it → s is its ego-motion from target frame to source frame.

Original: By combining these two steps, we can obtain the theoretically consistent multi-camera poses,which will further improve the scale-aware depth estimation accuracy.
Revised: By combining these two steps, we can obtain the theoretically consistent multi-camera poses, which will further improve the scale-aware depth estimation accuracy.

Original:  Experiments 
Revised:  Experiments 4.1 Experimental Setup

Original: 4.1 Experimental Setup

Original: We conduct experiments on both the Dense Depth for Automated Driving (DDAD) [27] andnuScenes datasets [46].
Revised: We conduct experiments on both the Dense Depth for Automated Driving (DDAD) [27] and nuScenes datasets [33].

Original: The basic backbones of depth and pose networks are the same as the Monodepth2 [28].
Revised: The basic backbones of depth and pose networks are the same as the Monodepth2 [28].

Original: We employ ResNet34 with ImageNet [47] pretrained weight as the encoder for allexperiments, including baseline methods.
Revised: We employ ResNet34 with ImageNet [47] pretrained weight as the encoder for all experiments, including baseline methods.

Original: Further, surrounding cameras have different focal lengthsand we refactor depth maps according to focal lengths following [48].
Revised: Further, surrounding cameras have different focal lengths and we refactor depth maps according to focal lengths following [48].

Original: In each scale, we adopted
Revised: In each scale, we adopted

Original: Z = 8 transformer layers and all features were downsampled to 20 × 12 and 20 × 11 before crossview attention on two datasets respectively.
Revised: Z = 8 transformer layers and all features were downsampled to 20 × 12 and 20 × 11 before crossview attention on two datasets respectively.

Original: Since DDAD dataset has self-occlusion areas, following[29], we annotate occlusion masks manually and use them to reweight photometric loss.
Revised: Since DDAD dataset has self-occlusion areas, following [29], we annotate occlusion masks manually and use them to reweight photometric loss.

Original: Moreimplementation details and visualizations are in supplementary materials.
Revised: More implementation details and visualizations are in supplementary materials.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 4.2 The Benchmark for Multi-camera Depth Estimation
Revised: 4.2 The Benchmark for Multi-camera Depth Estimation

Original: We compare our method with two existing state-of-the-art self-supervised monocular depth estimation methods (Monodepth2 [28] and PackNet-SfM [27]) and one multi-camera depth estimationmethod FSM [29].
Revised: We compare our method with two existing state-of-the-art self-supervised monocular depth estimation methods (Monodepth2 [28] and PackNet-SfM [27]) and one multi-camera depth estimation method FSM [29].

Original: Since FSM does not release code, its detailed experimental setting is unknown,such as evaluation, pretraining and hyperparameters.
Revised: Since FSM does not release code, its detailed experimental setting is unknown, such as evaluation, pretraining and hyperparameters.

Original: To fairly compare, we run all methods under the same training and evaluation settings.
Revised: To fairly compare, we run all methods under the same training and evaluation settings.

Original: Specifically, the input images are downsampled to640 × 384 and 640 × 352 resolutions.
Revised: Speciﬁcally, the input images are downsampled to 640 × 384 and 640 × 352 resolutions.

Original: Depth ground truth is generated by projecting LiDAR pointcloud to each view.
Revised: Depth ground truth is generated by projecting LiDAR point cloud to each view.

Original: Since Monodepth2 and PackNet-SfM are not designed for scale-aware depthestimation, we perform per-image median ground truth scaling [11] and post-processing [42] during evaluation.
Revised: Since Monodepth2 and PackNet-SfM are not designed for scale-aware depth estimation, we perform per-image median ground truth scaling [11] and post-processing [43] during evaluation.

Original: Moreover, SfM pretraining and joint pose estimation are designed for scale-awaredepth estimation, we do not adopt them in this experiment.
Revised: Moreover, SfM pretraining and joint pose estimation are designed for scaleaware depth estimation, we do not adopt them in this experiment.

Original: As shown in Tables 1 and 2, our methodachieves state-of-the-art performance, which demonstrates the effectiveness of the proposed crossview transformer.
Revised: As shown in Tables 1 and 2, our method achieves state-of-the-art performance. Our method outperforms baseline method Monodepth2, which demonstrates that our proposed cross-view approach improves the results since we fully entangle surrounding views.

Original: We test inference time of one batch (6 images) on a single RTX 3090:
Revised: We test inference time of one batch (6 images) on a single RTX 3090:

Original: Monodepth2 (0.028s), PackNet-SfM (0.471s), SurroundDepth (0.088s).
Revised: Monodepth2 (0.028s), PackNet-SfM (0.471s), SurroundDepth (0.088s).

Original: 4.3 Scale-aware Self-supervised Depth Estimation
Revised: 4.3 Scale-aware Self-supervised Depth Estimation

Original: Different from monocular depth estimation, once we get the extrinsic matrices of each camera anduse spatial context, the absolute scale of the world can be obtained.
Revised: Different from monocular depth estimation, once we get the extrinsic matrices of each camera and use spatial context, the absolute scale of the world can be obtained.

Original: Tables 3 and 4 show the resultswithout median-scaling at test time.
Revised: Tables 3 and 4 show the results without median-scaling at test time.

Original: We do not compare other methods since these methods cannot predict scale-aware depth maps.
Revised: We do not compare other methods since these methods cannot predict scale-aware depth maps.

Original: In contrast to the statement in [29], we find that the networkdirectly utilizing spatial photometric loss cannot force the network to generate scale-aware depths.
Revised: In contrast to the statement in [29], we ﬁnd that the network directly utilizing spatial photometric loss cannot force the network to generate scale-aware depths.

Original: Without pretraining, the scale of initial depth is greatly different from the ground truth, which leadsto invalid spatial photometric loss.
Revised: Without pretraining, the scale of initial depth is greatly different from the ground truth, which leads to invalid spatial photometric loss.

Original: With our proposed SfM pretraining, the depth network learns theabsolute scale and spatial context becomes effective.
Revised: With our proposed SfM pretraining, the depth network learns the absolute scale and spatial context becomes effective.

Original: The joint pose estimation further boosts theperformance.
Revised: The joint pose estimation further boosts the performance.

Original: We further evaluate the depth consistency between surrounding views.
Revised: We further evaluate the depth consistency between surrounding views.

Original: Specifically, we project eachview’s depth map to its neighboring views with extrinsic matrices and calculate depth errors.
Revised: Speciﬁcally, we project each view’s depth map to its neighboring views with extrinsic matrices and calculate depth errors.

Original: Table 8shows that the scale-ambiguous methods such as FSM and Monodepth2 cannot achieve multi-viewconsistency since the depth maps are scale-ambiguous but extrinsic matrices are scale-aware.
Revised: Tableshows that the scale-ambiguous methods such as FSM and Monodepth2 cannot achieve multi-view consistency since the depth maps are scale-ambiguous but extrinsic matrices are scale-aware.

Original: Withthe cross-view interaction, we can further boost the consistency.
Revised: With the cross-view interaction, we can further boost the consistency.

Original: [Table]
Revised: [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: 4.4 Ablation Study
Revised: 4.4 Ablation Study

Original: In this subsection
Revised: In this subsection

Original: [Table]
Revised: [Table]

Original: we conduct ablation studies to verify the effectiveness of each module in ourframework.
Revised: we conduct ablation studies to verify the effectiveness of each module in our framework.

Original: All experiments are conducted on DDAD dataset.
Revised: All experiments are conducted on DDAD dataset.

Original: Cross-view Transformer:
Revised: Cross-view Transformer:

Original: Table 5 shows that both skip connection and multi-scale formulationcontribute to the final results.
Revised: Table 5 shows that both skip connection and multi-scale formulation contribute to the ﬁnal results.

Original: Skip connections are able to alleviate the information loss caused bythe downsampling of feature maps.
Revised: Skip connections are able to alleviate the information loss caused by the downsampling of feature maps.

Original: The multi-scale cross-view transformer helps the network tobetter learn both fine-grained and global features.
Revised: The multi-scale cross-view transformer helps the network to better learn both ﬁne-grained and global features.

Original: Joint Pose Estimation:
Revised: Joint Pose Estimation:

Original: Table 6 shows separate pose estimation cannot guarantee the multi-camerapose consistency.
Revised: Table 6 shows separate pose estimation cannot guarantee the multi-camera pose consistency.

Original: To provide a straightforward baseline (second line), we project the ego-motion ofall views to the front view and add a loss between the projected and predicted ego-motion of the frontview.
Revised: To provide a straightforward baseline (second line), we project the ego-motion of all views to the front view and add a loss between the projected and predicted ego-motion of the front view.

Original: However, this method gets marginal improvements.
Revised: However, this method gets marginal improvements.

Original: We note that our joint pose estimationmethod gets comparable performances with ground-truth pose experiment.
Revised: We note that our joint pose estimation method gets comparable performances with ground-truth pose experiment.

Original: This result indicates thatground truth ego-motion may have noise, even for carefully-calibrated public datasets.
Revised: This result indicates that ground truth ego-motion may have noise, even for carefully-calibrated public datasets.

Original: Thus, it isnecessary to predict accurate scale-aware poses.
Revised: Thus, it is necessary to predict accurate scale-aware poses.

Original: Scale-aware SfM Pretraining: Table 7 shows that region mask is able to help our model to learnreal-world scale.
Revised: Scale-aware SfM Pretraining: Table 7 shows that region mask is able to help our model to learn real-world scale.

Original: If we do not set region mask for SIFT matching, we will get wrong correspondences and the depth network is not able to learn real-world scale.
Revised: If we do not set region mask for SIFT matching, we will get wrong correspondences and the depth network is not able to learn real-world scale.

Original: By filtering out the correspondences far from the epipolar lines, we will further boost the performance of the depth network.
Revised: By ﬁltering out the correspondences far from the epipolar lines, we will further boost the performance of the depth network.

Original:  Limitations
Revised:  Limitations

Original: Although the proposed SurroundDepth can boost the multi-view depth consistency with scale-awaretraining strategy and cross-view transformers (as shown in Table 8), our method cannot theoreticallyguarantee this consistency.
Revised: Although the proposed SurroundDepth can boost the multi-view depth consistency with scale-aware training strategy and cross-view transformers (as shown in Table 8), our method cannot theoretically guarantee this consistency.

Original: As future work, we will directly predict voxel occupancy of the 3D spaceinstead of each view’s depth maps.
Revised: As future work, we will directly predict voxel occupancy of the 3D space instead of each view ’s depth maps.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we propose SurroundDepth for self-supervised multi-camera depth estimation.
Revised: In this paper, we propose SurroundDepth for self-supervised multi-camera depth estimation.

Original: Thecore insight of our method is to entangle multi-camera information and jointly process all surrounding views.
Revised: The core insight of our method is to entangle multi-camera information and jointly process all surrounding views.

Original: The cross-view transformer is performed at multiple scales to incorporate multi-viewfeatures.
Revised: The cross-view transformer is performed at multiple scales to incorporate multi-view features.

Original: To obtain scale-aware depth predictions, we present Structure-from-Motion pretrainingand joint pose estimation to fully leverage multi-camera extrinsic matrices.
Revised: To obtain scale-aware depth predictions, we present Structure-from-Motion pretraining and joint pose estimation to fully leverage multi-camera extrinsic matrices.

Original: Our method achievesstate-of-the-art performance on the multi-camera depth estimation datasets. 
Revised: Our method achieves state-of-the-art performance on the multi-camera depth estimation datasets. Acknowledgments

Revised: This work was supported in part by the National Key Research and Development Program of China under Grant 2017YFA0700802, in part by the National Natural Science Foundation of China under

Revised: Grant 62125603 and Grant U1813218, in part by a grant from the Beijing Academy of Artiﬁcial

Revised: Intelligence (BAAI).