DPSN ET : E ND - TO - END D EEP P LANE S WEEP S TEREO
A BSTRACT
Multiview stereo aims to reconstruct scene depth from images acquired by a camera under arbitrary motion.
Recent methods address this problem through deep learning, which can utilize semantic cues to deal with challenges such as textureless and reﬂective regions.
In this paper, we present a convolutional neural network called DPSNet (Deep Plane Sweep Network) whose design is inspired by best practices of traditional geometry-based approaches for dense depth reconstruction.
Rather than directly estimating
depth and/or optical ﬂow correspondence from image pairs as done in many previous deep learning methods
, DPSNet takes a plane sweep approach that involves building a cost volume from deep features using the plane sweep algorithm, regularizing the cost volume via a contextaware cost aggregation, and regressing the dense depth map from the cost volume.
The cost volume is constructed using a differentiable warping process that allows for end-to-end training of the network.
Through the effective incorporation of conventional multiview stereo concepts within a deep learning framework, DPSNet achieves state-of-the-art reconstruction results on a variety of challenging datasets.
I NTRODUCTION
Various image understanding tasks, such as semantic segmentation Couprie et al.
and
human pose/action recognition Shotton et al.
; Wang et al.
(2016), have been shown to beneﬁt from 3D scene information.
A common approach to reconstructing 3D geometry is by multiview stereo, which infers depth based on point correspondences among a set of unstructured images Hartley &
Zisserman (2003);
Sch ¨ onberger et al.
To solve for these correspondences, conventional techniques employ photometric consistency constraints on local image patches.
Such photo-consistency constraints, though effective in many instances, can be unreliable in scenes containing textureless and reﬂective regions.
Recently, convolutional neural networks (CNNs) have demonstrated some capacity to address this issue by leveraging semantic information inferred from the scene.
The most promising of these methods employ a traditional stereo matching pipeline, which involves computation of matching cost volumes, cost aggregation, and disparity estimation Flynn et al.
; Kendall et al.
Huang et al.
; Chang & Chen (2018).
Some are designed for binocular stereo Ummenhofer et al.
; Kendall et al.
(2017); Chang & Chen (2018) and cannot readily be extended to multiple views.
The CNN-based techniques for multiview processing Flynn et al.
; Huang et al.
both follow the plane-sweep approach, but require plane-sweep volumes as input to their networks.
As a result, they are not end-to-end systems that can be trained from input images to disparity maps.
In this paper, we present Deep Plane Sweep Network (DPSNet), an end-to-end CNN framework for robust multiview stereo.
In contrast to previous methods that employ the plane-sweep approach Huang et al.
; Flynn et al.
, DPSNet fully models the plane-sweep process, including construction of plane-sweep cost volumes, within the network.
This is made possible through the use of a differentiable warping module inspired by spatial transformer networks Jaderberg et al.
(2015) to build the cost volumes.
With the proposed network, plane-sweep stereo can be learned in an end-to-end fashion.
Additionally, we introduce a cost aggregation module based on [Figure] local cost-volume ﬁltering Rhemann et al.
(2011) for context-aware reﬁnement of each cost slice.
Through this cost-volume regularization, the effects of unreliable matches scattered within the cost volume are reduced considerably.
With this end-to-end network for plane-sweep stereo and the proposed cost aggregation, we obtain state-of-the-art results over several standard datasets.
Ablation studies indicate that each of these technical contributions leads to appreciable improvements in reconstruction accuracy.
R ELATED W ORK
CNN-based depth estimation has been studied for stereo matching, depth from single images, and multiview stereo.
Recent work in these areas are brieﬂy reviewed in the following.
Stereo matching Methods for stereo matching address the particular case of depth estimation where the input is a pair of rectiﬁed images captured by a stereo rig.
Various network structures have been introduced for this problem.
Zbontar & LeCun (2016) present a Siamese network structure to compute matching costs based on the similarity of two image patches.
The estimated initial depth is then reﬁned by traditional cost aggregation and reﬁnement as post-processing.
Mayer et al.
(2016) directly stack several convolution and deconvolution layers upon the matching costs and train the network to minimize the distance between the estimates and ground truth.
Liang et al.
(2018) propose a CNN that estimates initial disparity and then reﬁnes it using both prior and posterior feature consistency in an end-to-end manner.
Kendall et al.
(2017) leverage geometric knowledge in building a cost volume from deep feature representations.
It also enables learning of contextual information in a 3D volume and regresses disparity in an end-to-end manner.
Chang & Chen (2018) introduce a pyramid pooling module for incorporating global contextual information into image features and a stacked hourglass 3D CNN to extend the regional support of contextual information.
Depth from single images
Similar to these stereo matching approaches, single-image methods extract CNN features to infer scene depths and perform reﬁnements to increase depth accuracy.
The ﬁrst of these methods was introduced by Eigen et al.
(2014), which demonstrated that CNN features could be utilized for depth inference.
Later, Liu et al.
(2016) combined a superpixel-based conditional random ﬁeld (CRF) to a CNN to improve the quality of depth estimates from single images.
To facilitate training, recent studies Zhou et al.
; Wang et al.
; Mahjourian et al.
; Yin & Shi (2018) present an end-to-end learning pipeline that utilizes the task of view synthesis as supervision for single-view depth and camera pose estimation.
These systems consist of a depth network and a pose estimation network which simultaneously train on sequential images with a loss computed from images warped to nearby views using the estimated depth.
View synthesis has similarly been used as supervision by warping between stereo image pairs Garg et al.
(2016); Godard et al.
In contrast to these single-image works which employ warping as a component of view synthesis for self-supervised learning, our network computes warps with respect to multiple depth planes to produce plane-sweep cost volumes both for training and at test time.
The cost volumes undergo further processing in the form of cost aggregation and regularization to improve the robustness of depth estimates.
Multi-view stereo In multi-view stereo, depth is inferred from multiple input images acquired from arbitrary viewpoints.
To solve this problem, some methods recover camera motion between the unstructured images but are designed to handle only two views Ummenhofer et al.
; Li et al.
The DeMoN system Ummenhofer et al.
(2017) consists of encoder-decoder networks for optical ﬂow, depth/motion estimation, and depth reﬁnement.
By alternating between estimating optical ﬂow and depth/motion, the network is forced to use both images in estimating depth, rather than resorting to single-image inference.
Li et al.
(2018) perform monocular visual odometry in an unsupervised manner.
In the training step, the use of stereo images with extrinsic parameters allows 3D depth estimation to be estimated with metric scale.
Among networks that can handle an arbitrary number of views, camera parameters are assumed to be known or estimated by conventional geometric methods.
Ji et al.
(2017) introduce an endto-end learning framework based on a viewpoint-dependent voxel representation which implicitly encodes images and camera parameters.
The voxel representation restricts the scene resolution that can be processed in practice due to limitations in GPU memory.
Im et al.
(2018b) formulate a geometric relationship between optical ﬂow and depth to reﬁne the estimated scene geometry, but is designed for image sequences with a very small baseline, i.e., an image burst from a handheld camera.
Huang et al.
(2018) compute a set of plane-sweep volumes using calibrated pose data as input for the network, which then predicts an initial depth feature using an encoder-decoder network.
In the depth prediction step, they concatenate a reference image feature to the decoder input as an intra-feature aggregation, and cost volumes from each of the input images are aggregated by max-pooling to gather information for the multiview matching.
Its estimated depth map is reﬁned using a conventional CRF.
By contrast, our proposed DPSNet is developed to be trained end-to-end from input images to the depth map.
Moreover, it leverages conventional multiview stereo concepts by incorporating context-aware cost aggregation.
Finally, we would like to refer the reader to the concurrent work by Yao et al.
(2018) that also adopts differential warping to construct a multi-scale cost volume, then reﬁned an initial depth map guided by a reference image feature.
Our work is independent of this concurrent effort.
Moreover, we make distinct contributions: (1) We focus on dense depth estimation for a reference image in an end-to-end learning manner, different from
Yao et al.
(2018) which reconstructs the full 3D of objects.
(2) Our cost volume is constructed by concatenating input feature maps, which enables inference of accurate depth maps even with only two-view matching.
(3) Our work reﬁnes every cost slice by applying context features of a reference image, which is beneﬁcial for alleviating coarsely scattered unreliable matches such as for large textureless regions.
A PPROACH
Our Deep Plane Sweep Network (DPSNet) is inspired by traditional multiview stereo practices for dense depth estimation and consists of four parts: feature extraction, cost volume generation, cost aggregation and depth map regression.
The overall framework is shown in ﬁgure 2.
3. M ULTI - SCALE F EATURE E XTRACTION
We ﬁrst pass a reference image and target images through seven convolutional layers ( 3 × 3 ﬁlters except for the ﬁrst layer, which has a 7 × 7 ﬁlter) to encode them, and extract hierarchical contextual information from these images using a spatial pyramid pooling (SPP) module He et al.
(2014) with four ﬁxed-size average pooling blocks ( 16 × 16 , 8 × 8 , 4 × 4 , 2 × 2 ).
The multi-scale features extracted by SPP have been shown to be effective in many visual perception tasks such as visual recognition He et al.
(2014), scene parsing Zhao et al.
(2017) and stereo matching Huang et al.
After upsampling the hierarchical contextual information to the same size as the original feature map, we concatenate all the feature maps and pass them through 2D convolutional layers.
This process yields 32-channel feature representations for all the input images, which are next used in building cost volumes.
3. C OST V OLUME G ENERATION USING U NSTRUCTURED T WO -V IEW I MAGES
We propose to generate cost volumes for the multiview images by adopting traditional plane sweep stereo Collins (1996); Yang & Pollefeys (2003); Ha et al. ; Im et al. (2018a), originally devised for dense depth estimation.
The basic idea of plane sweep stereo is to back-project the image set onto successive virtual planes in the 3D space and measure photo-consistency among the warped images for each pixel.
In a similar manner to traditional plane sweep stereo, we construct a cost [Figure] volume from an input image pair.
To reduce the effects of image noise, multiple images can be utilized by averaging cost volumes for other pairs.
For this cost volume generation network, we ﬁrst set the number of virtual planes perpendicular to the z -axis of the reference viewpoint [0 , 0 , 1] (cid:124) and uniformly sample them in the inverse-depth space as follows:
[Equation]
where L is the total number of depth labels and d min is the minimum scene depth as speciﬁed by the user.
Then, we warp all the paired features F i , ( i = 1 , .., N ), where i is an index of viewpoints and N is the total number of input views, into the coordinates of the reference feature (of size Width ×
Height × CHannel ) using pre-computed intrinsics K and extrinsic parameters consisting of a rotation matrix R i and a translation matrix t i of the i th camera:
[Equation]
where u, ˜ u l are the homogeneous coordinates of a pixel in the reference view and the projected coordinates onto the paired view, respectively.
˜ F il ( u ) denotes the warped features of the paired image through the l th virtual plane.
Unlike the traditional plane sweeping method which utilizes a distance metric, we use a concatenation of features in learning a representation and carry this through to the cost volume as proposed in Kendall et al.
We obtain a 4D volume ( W × H × 2 CH × L ) by concatenating the reference image features and the warped image features for all of the depth labels.
In Eq.
(2), we assume that all images are captured by the same camera, but it can be directly extended to images with different intrinsics.
For the warping process, we use a spatial transformer network Jaderberg et al.
(2015) for all hypothesis planes, which does not require any learnable parameters.
In Table 3, we ﬁnd that concatenating features improves performance over the absolute difference of the features.
Given the 4D volume 1 , our DPSNet learns a cost volume generation of size W × H × L by using a series of 3D convolutions on the concatenated features.
All of the convolutional layers consist of 3 × 3 × 3 ﬁlters and residual blocks.
In the training step, we only use one paired image (while the other is the reference image) to obtain the cost volume.
In the testing step, we can use any number of paired images ( N ≥ 1 ) by averaging all of the cost volumes.
3. C OST A GGREGATION
The key idea of cost aggregation Rhemann et al.
(2011) is to regularize the noisy cost volume through edge-preserving ﬁltering He et al.
within a support window.
Inspired by traditional cost volume ﬁltering, we introduce a context-aware cost aggregation method in our end-to-end learning {+
+}
We implement a 5D volume that includes a batch dimension.
process. The context network takes each slice of the cost volume and the reference image features extracted from the previous step, and then outputs the reﬁned cost slice.
We run the same process for all the cost slices.
The ﬁnal cost volume is then obtained by adding the initial and residual volumes as shown in ﬁgure 2 (top).
Here, we use dilated convolutions in the context network for cost aggregation to better exploit contextual information Chen et al.
; Yu & Koltun (2016).
The context network consists of seven convolutional layers with 3 × 3 ﬁlters, where each layer has a different receptive ﬁeld (1, 2, 4, 8, 16, 1, and 1).
We jointly learn all the parameters, including those of the context network.
All cost slices are processed with shared weights of the context network.
Then, we upsample the cost volume, whose size is equal to the feature size, to the original size of the images via bilinear interpolation.
We ﬁnd that this leads to moderate performance improvement as shown in Table 3.
3. D EPTH R EGRESSION
We regress continuous depth values using the method proposed in Kendall et al.
The probability of each label l is calculated from the predicted cost c l via the softmax operation σ ( · ) .
The predicted label ˆ l is computed as the sum of each label l weighted by its probability.
With the predicted label, the depth is calculated from the number of labels L and minimum scene depth d min as follows:
[Equation]
We set L and d min to 64 and 0.5, respectively.
3. T RAINING L OSS
Let θ be the set of all the learnable parameters in our network, which includes feature extraction, cost volume generation and cost aggregation (plane sweep and depth regression have no learnable parameters).
Let ˆ d, ˜ d denote the predicted depth from the initial and reﬁned cost volumes, respectively, and let d gt be the corresponding supervision signal.
The training loss is then formulated as
[Equation]
where | · | H denotes the Huber norm, referred to as SmoothL1Loss in PyTorch.
The weight value λ for depth from the initial cost volume is set to 0.7.
E XPERIMENTS 4. I MPLEMENTATION DETAILS
In the training procedure, we use image sequences, ground-truth depth maps for reference images, and the provided camera poses from public datasets, namely SUN3D, RGBD, and Scenes11 2 .
We train our model from scratch for 1200K iterations in total.
All models were trained end-to-end with the ADAM optimizer ( β 1 = 0.9, β 2 = 0.999).
We use a batch size of 16 and set the learning rate to 2 e − 4 for all iterations.
The training is performed with a customized version of PyTorch on an
Nvidia 1080 GPU, which usually takes four days.
4. C OMPARISON WITH STATE - OF - THE - ART METHODS
In our evaluations, we use common quantitative measures of depth quality: absolute relative error (Abs Rel), absolute relative inverse error (Abs R-Inv), absolute difference error (Abs diff), square relative error (Sq Rel), root mean square error and its log scale (RMSE and RMSE log) and inlier ratios ( δ < 1 .
25 i where i ∈ { 1 , 2 , 3 } ).
All are standard metrics used in a public benchmark suite 3 .
[Table] [Table] For our comparisons, we choose state-of-the-art methods for traditional geometry-based multiview stereo (COLMAP) Sch ¨ onberger & Frahm (2016), depth from unstructured two-view stereo (DeMoN) Ummenhofer et al.
and CNN-based multiview stereo (DeepMVS) Huang et al.
We estimate the depth maps from two unstructured views using the test sets in MVS, SUN3D, RGBD and Scenes11, as done for DeMoN 4 .
The results are reported in Table 1.
Our DPSNet provides the best performance on nearly all of the measures.
Of particular note, DPSNet accurately recovers scene depth in homogeneous regions as well as along object boundaries as exhibited in ﬁgure 3.
DeMoN generally produces good depth estimates but often fails to reconstruct scene details such as the keyboard (third row) and ﬁne structures (ﬁrst, second and fourth rows).
By contrast, DPSNet estimates accurate depth maps at those regions because the differential feature warping penalizes inaccurate reconstructions, playing a role similar to the left-right consistency check that has been used in stereo matching Garg et al.
The ﬁrst and third rows of ﬁgure 3 exhibit problems of COLMAP and DeepMVS in handling textureless regions.
DPSNet instead produces accurate results, courtesy of the cost aggregation network.
For a more balanced comparison, we adopt measures used in Huang et al.
(2018) as additional evaluation criteria: (1) completeness, which is the percentage of pixels whose errors are below a certain threshold.
(2) geometry error, taking the L1 distance between the estimated disparity and the ground truth.
(3) photometry error, which is the L1 distance between the reference image and warped image using the estimated disparity map.
The results for COLMAP, DeMoN and DeepMVS are directly reported from Huang et al.
(2018) in Table 2.
In this experiment, we use the ETH3D dataset on which all methods are not trained.
Following Yao et al.
(2018), we take 5 images with 1152 × 864 resolution and set 192 depth labels based on ground-truth depth to obtain optimal results for MVSNet.
For the DPSNet results, we use 4 views with 810 × 540 resolution and set 64 labels whose range is determined by the minimum depth values of the ground truth.
In Table 2, our DPSNet shows the best performance overall among the all the comparison methods, except for ﬁltered COLMAP.
Although ﬁltered COLMAP achieves the best performance, its completeness is only 71% and its unﬁltered version shows a signiﬁcant performance drop in all error metrics.
On the other hand, our DPSNet with 100% completeness shows promising results on all [Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure]
[Figure]
[Figure]
[-δ < 1 . 25δ < 1 . 
 [Figure] [Figure] [Figure]-]Ours 0.
measures.
We note that our DPSNet has a different purpose compared to COLMAP and MVSNet. COLMAP and MVSNet are designed for full 3D reconstruction with an effective outlier rejection process, while DPSNet aims to estimate a dense depth map for a reference view.
4. A BLATION S TUDY
An extensive ablation study was conducted to examine the effects of different components on DPSNet performance.
We summarize the results in Table 3.
Cost Volume Generation In Table 3 (a) and (e), we compare the use of cost volumes generated using the traditional absolute difference Collins (1996) and using the concatenation of features from the reference image and warped image.
The absolute difference is widely used for depth label selection via a winner-take-all strategy.
However, we observe that feature concatenation provides [Figure] [Figure] better performance in our network than the absolute difference.
A possible reason is that the CNN may learn to extract 3D scene information from the tensor of stacked features.
The tensor is fed into the CNN to produce an effective feature for depth estimation, which is then passed through our cost aggregation network for the initial depth reﬁnement.
Cost Aggregation For our cost aggregation sub-network, we compare DPSNet with and without it in Table 3 (e) and (b), respectively.
It is shown that including the proposed cost aggregation leads to signiﬁcant performance improvements.
Examples of depth map reﬁnement with the cost aggregation are displayed in ﬁgure 5.
Our cost aggregation is also compared to using a stacked hourglass to aggregate feature information along the depth dimension as well as the spatial dimensions as done recently for stereo matching Chang & Chen (2018).
Although the stacked hourglass is shown in Table 3 (c) to enhance depth results, its improvement is smaller than ours, which uses reference image features to guide the aggregation.
ﬁgure 5 (a), (b) show examples of the depth map results before and after our cost aggregation.
It demonstrates that the cost aggregation sub-network outputs more accurate depth, especially on homogeneous regions.
[Table]
Datasets: MVS, SUN3D, RGBD, Scenes11.
For further analysis of cost aggregation, we display slices of 3D cost volumes after the softmax operation (in Eq.
that span depth labels and the rows of the images.
The cost slices in ﬁgure 5 (c), (d) show that our feature-guided cost aggregation regularizes noisy cost slices while preserving edges well.
The cleaner cost proﬁles that ensue from the cost aggregation lead to clearer and edge-preserving depth regression results.
As mentioned in a recent study Hu & Mordohai (2012), a cost proﬁle that gives conﬁdent estimates should have a single, distinct minimum (or maximum), while an ambiguous proﬁle has multiple local minima or multiple adjacent labels with similar costs, making it hard to exactly localize the global minimum.
Based on two quantitative conﬁdence measures Hu & Mordohai (2012) on cost volumes in Table 4, the proposed aggregation improves the reliability of the correct match corresponding to the minimum cost.
Depth Label Sampling
In the plane sweep procedure, depth labels can be sampled in either the depth domain or the inverse depth domain, which provides denser sampling in areas closer to a camera.
Table 3 (d) and (e) show that uniform depth label sampling in the inverse depth domain produces more accurate depth maps in general.
Number of Images We examine the performance of DPSNet with respect to the number of input images.
As displayed in ﬁgure 6a, a greater number of images yields better results, since cost volume
[Figure] [Figure] [Figure] noise is reduced through averaging over more images, and more viewpoints help to provide features from areas unseen in other views.
ﬁgure 6 shows that adding input views aids in distinguishing object boundaries.
Note that the performance improvement plateaus when seven or more images are used.
Rectiﬁed Stereo Pair CNNs-based stereo matching methods have similarity to DPSNet, but differ from it in that correspondences are obtained by shifting learned features in Mayer et al.
Kendall et al.
(2017); Tulyakov et al.
The purpose of this study is to show readers that not only descriptor shift but also plane sweeping can be applied to rectiﬁed stereo matching.
We apply
DPSNet on the KITTI dataset, which provides rectiﬁed stereo pairs with a speciﬁc baseline.
As shown in ﬁgure 7, although DPSNet is not designed to work on rectiﬁed stereo images, it produces reasonable results.
In particular, DPSNet ﬁne-tuned on the KITTI dataset in Table 5 achieves performance similar to Mayer et al.
in terms of D1-all score, with 4.34% for all pixels and 4.05% for non-occluded pixels in the KITTI benchmark.
We expect that the depth accuracy would improve if we were to adopt rectiﬁed stereo pair-speciﬁc strategies, such as the feature consistency check in Liang et al.
D ISCUSSION
We developed a multiview stereo network whose design is inspired by best practices of traditional non-learning-based techniques.
The plane sweep algorithm is formulated as an end-to-end network via a differentiable construction of plane sweep cost volumes and by solving for depth as a multilabel classiﬁcation problem.
Moreover, we propose a context-aware cost aggregation method that leads to improved depth regression without any post-processing.
With this incorporation of traditional multiview stereo schemes into a deep learning framework, state-of-the-art reconstruction results are achieved on a variety of datasets.
Directions exist for improving DPSNet.
One is to integrate semantic instance segmentation into the cost aggregation, similar to the segment-based cost aggregation method of Mei et al.
Another direction is to improve depth prediction by employing viewpoint selection in constructing cost volumes Gallup et al.
; Sch¨onberger et al.
(2016), rather than by simply averaging the estimated cost volumes as currently done in DPSNet.
Lastly, the proposed network requires pre-calibrated intrinsic and extrinsic parameters for reconstruction.
Lifting this restriction by additionally estimating camera poses in an end-to-end learning framework is an important future challenge. A CKNOWLEDGEMENT
This work was supported by the Technology Innovation Program (No. 2017-10069072) funded
By the Ministry of Trade, Industry & Energy (MOTIE, Korea), and supported in part by Air
Force Research Laboratory (AFRL) project FA23861714660.
Sunghoon Im was partially supported by Global Ph.D. Fellowship Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2016907531).
Hae-Gon Jeon was partially supported by
Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (2018R1A6A3A03012899).