ElasticMVS: Learning elastic part representation for self-supervised multi-view stereopsis
Abstract
Self-supervised multi-view stereopsis (MVS) attracts increasing attention for learning dense surface predictions from only a set of images without onerous groundtruth 3D training data for supervision.
However, existing methods highly rely onthe local photometric consistency, which fail to identify accurately dense correspondence in broad textureless or reflectant areas.
In this paper, we show that geometric proximity such as surface connectedness and occlusion boundaries implicitly in- ferred from images could serve as reliable guidance for pixel-wise multi-viewcorrespondences.
With this insight, we present a novel elastic part representation,which encodes physically-connected part segmentations with elastically-varyingscales, shapes and boundaries.
Meanwhile, a self-supervised MVS frameworknamely ElasticMVS is proposed to learn the representation and estimate per-viewdepth following a part-aware propagation and evaluation scheme.
Specifically, thepixel-wise part representation is trained by a contrastive learning-based strategy,which increases the representation compactness in geometrically concentrated areasand contrasts otherwise.
ElasticMVS iteratively optimizes a part-level consistencyloss and a surface smoothness loss, based on a set of depth hypotheses propagated from the geometrically concentrated parts.
Extensive evaluations convey thesuperiority of ElasticMVS in the reconstruction completeness and accuracy, aswell as the efficiency and scalability.
Particularly, for the challenging large-scalereconstruction benchmark, ElasticMVS demonstrates significant performance gainover both the supervised and self-supervised approaches.
Introduction
Given a set of images with known camera parameters, multi-view stereopsis (MVS) aims to reconstruct the dense and accurate geometry of the scene.
Existing learning-based approaches (Yaoet al., 2018), typically being cast as an end-to-end depth regression task, outperform the traditionalgeometry-based ones (Schönberger et al., 2016).
However, their generalization ability in open environments is severely hindered by the availability of the onerous laser-scanned 3D training data.
Self-supervised MVS (Khot et al., 2019; Dai et al., 2019) lifts such limitation by leveraging themulti-view photometric consistency in place of the supervisory signals.
Unfortunately, the pixel-levelphotometric regularization is susceptible to textureless patterns and illumination variations, leadingto incomplete and inaccurate reconstructions, especially in outdoor environments.
In this paper, we tackle the above challenges with a novel image parsing paradigm.
Instead oftreating images as projections of 3D scenes, we show that the inherent geometric correlations, suchas surface connectedness, smoothness and boundaries, could be inferred implicitly from images andserve as a reliable guidance for pixel-wise multi-view correspondence estimation.
The mainstreamsolutions of feature-based regularizations introduce either hand-crafted structures such as lines and planes from superpixels (Romanoni and Matteucci, 2019; Bódis-Szomorú et al., 2014; Micusíkand Kosecka, 2010), or semantic features pre-trained from limited geometric-unrelated labels (Xuet al., 2021; Hung et al., 2019).
Different from those, we aim to build the correlation betweenthe latent space of the representation and the scene space of the surface in a self-supervised way.
With this insight, we present a novel elastic part representation which encodes physically-connectedpart segmentations with elastically-varying scales, shapes and boundaries.
Unlike the traditionalsegmentation networks with a fixed number of labels, our pixel-wise part representation directlyreflects the surface correlation, i.e., the more similar the two representations are, the higher probabilitythey belong to the same physical surface part.
Meanwhile, a self-supervised MVS framework namely
ElasticMVS is proposed to learn the representation and estimate per-view depth iteratively followinga part-aware propagation and evaluation scheme.
We believe that this framework provides a neatsolution to bridge the gap between the multi-view stereopsis and the point cloud reconstruction (Bergeret al., 2014), which has the potential to be effectively exploited by many other MVS methods.
Benefiting from the elastic part representation, the inherent geometric correlations are effectivelyexploited by a patchmatch-based propagation strategy (Galliani et al., 2015).
Specifically, in eachiteration, depth is estimated by selecting the best depth hypotheses adaptively propagated only fromthe geometrically concentrated part.
Different from other adaptive propagation strategy (Wanget al., 2021a; Xu and Tao, 2019) based only on pure image features without explicit geometricconstraints, our part-aware propagation scheme directly relies on physically plausible indicators.
Besides, the pixel-level photometric consistency loss is enhanced by the constraints of feature-levelcorrespondence and non-local piecewise surface smoothness, which provide part-aware matchingcues to guide a more robust depth prediction in textureless and reflectant areas.
To train the elastic partrepresentation, a contrastive learning-based strategy is implemented by increasing the representationcompactness in geometrically concentrated areas and contrasts otherwise.
Without the ground-truth3D information, these geometrically concentrated areas are identified by the depth map generatedfrom the initialized ElasticMVS without the part representation, which owns the same workflow asthe traditional geometry-based stereopsis (Schönberger et al., 2016).
A noise-robust training loss isimplemented to get rid of undesirable identifications.
Experiments on ‘DTU’ and ‘Tanks and Temples (T&T)’
datasets demonstrate that the proposed
ElasticMVS significantly outperforms all state-of-the-art self-supervised learning-based methods.
Remarkably, in the challenging large-scale reconstruction benchmark (T&T advanced), ElasticMVSshows dramatic performance improvements w.r.t.
both the supervised and self-supervised approaches.
Besides, the systematic evaluations indicate that our elastic part representation is superior in modelingelastically-varied parts, while being robust and lightweight in training and inference.
Related work2.1 Multi-view Stereopsis (MVS)
The major works in multi-view stereopsis focus on the view-wise depth map estimation (Tola et al.
,2012; Furukawa and Ponce, 2010) and then fuse together to form the point cloud(Zach, 2008; Goeseleet al., 2007).
Among these methods, patchmatch-based depth optimization methods(Galliani et al.
,2015; Schönberger et al., 2016; Xu and Tao, 2019), initialized from the randomly sampled depthhypotheses, show great run-time and memory efficiency in depth map estimation.
Some learningbased methods (Wang et al., 2021a; Lee et al., 2021) introduce an iterative coarse-to-fine optimizationin an end-to-end trainable architecture to improve each of the core steps in patchmatch.
However,these algorithms suffer from incomplete and noisy depth predictions in textureless and reflectant areas.
Several previous works in MVS solved the problem by introducing pre-defined global structural priorssuch as planes to resolve the ambiguity. For instance, TAPAMVS (Romanoni and Matteucci, 2019)segments the whole image into a bunch of superpixels and assumes textureless areas piecewise planar. RANSAC is applied to estimate its corresponding plane parameters. Beyond that, a bunch of worksin point cloud reconstruction deal with reconstruction artifacts with outlier, noisy or missing data.
Forinstance, implicit representation (Kazhdan et al., 2006) is used to provide global surface smoothnessprior given noisy data, and exterior visibility (Katz et al., 2007) is used to detect and eliminateredundant data.
However, neither the handcrafted plane structure nor the complex smoothness priorcan provide sufficiently diverse and accurate surface part modeling for real-world scenarios.
Currently, learning methods based on 3D cost volume regularization demonstrate state-of-the-artresults in several benchmarks.
These methods optimizes the 3D geometry by 3D CNN, unprojectingfrom 2D images or features into the 3D volume (Ji et al., 2017, 2020; Kendall et al., 2017; Yao et al.
,2018; Kar et al., 2017; Yao et al., 2019;
Yi et al., 2019; Chen et al., 2019; Gu et al., 2019; Chenget al., 2020).
However, these supervised methods crucially rely on ground-truth 3D data, which arehard to generalize in wild open environments.
To realize self-supervised MVS without the onerousground-truth 3D training data, (Dai et al., 2019; Huang et al., 2021;
Khot et al., 2019; Zhang et al.
Xu et al., 2021; Liu et al., 2019; Niemeyer et al., 2020) replace the supervision signal withthe unsupervised rendering loss or cross-view feature consistency loss.
However, reprojection undermultiple views is highly sensitive to environmental illumination, which is hard to generalize in highlyvariable scenarios.
2.2 Part Representation
Recently, learning representations from the image in a self-supervised training strategy has gainedgrowing popularity in 2D vision tasks, which use contrastive learning (Wu et al., 2018;
Chen et al.
,2020b,a; Caron et al., 2020) to deal with the lack of ground-truth labels.
They use InfoNCE (Van den
Oord et al., 2018) to generate a unique embedding for each picture in a batch.
Wang et al.
(2021b) further leverages a dense projection head to generalize such embedding from image-level to patch-level,which is more suitable for performing dense prediction tasks such as semantic segmentations.
Hunget al.
; Wang et al.
(2020) assigns the equivariance constraint and the semantic consistency regularization on the learned part response representation
(Zhou et al., 2016) to predict part segmentationsthat are consistent across different instances or camera poses.
Xu et al.
(2021) incorporates learnedsemantic features into multi-view stereopsis pipeline to boost the reconstruction ability.
However,previous works only concentrate on generating semantic features and using it forward to performdown-stream tasks.
Instead, we utilize the rough depth map as supervision to the semantic learningprocess, which forms a natural integration of photometric and geometric information to provideelastic part representations.
Method
We estimate the depth maps of a given reference image x and the corresponding M unstructuredsource images with known camera calibrations.
The final point cloud reconstruction is obtained byfusing all of them.
Since the geometry information gained from any photometric loss contains missingdata and artifacts in some areas of the image (especially in textureless regions), a heuristic geometryregularization for depth prediction is unavailable in these regions.
Therefore, the key innovation in ourframework is a novel elastic part representation , which encodes sufficient geometric details to guide a piecewise-smooth depth map prediction.
We first formally define our part-aware representation in Sec. 3.1.
Based upon the learned representation, we introduce two improvements on depth hypothesespropagation and evaluation for part-aware patchmatch in Sec. 3.2.
We present the self-supervisedtraining scheme in Sec.
3.3 to robustly train the representation with noisy initialization.
3.1 Elastic Part Representation
Suppose that ( ˜ d, ˜ n ) ∈
R H
× W × 4 are the underlying depth and normal map of the image x , wefirst define the geometry-aware part segmentation S p related to each pixel location p ∈ R 2 ,which is geometrically concentrated inside: S p ≜ { q ∈ R 2 | Γ { ( ˜ d p , ˜ n p ); ( ˜ d q , ˜ n q ) } ≤ ϵ } , whereΓ { ( ˜ d p , ˜ n p ); ( ˜ d q , ˜ n q ) } ≜ | ([ q, ˜ d q ] − [ p, ˜ d p ]) ·
˜ n p | is the 3D point-to-plane distance (Tateno et al., 2015)between two pixels.
Equipped with the part segmentation S p for each pixel p , we propose to learn a per-pixel elastic partrepresentation z p ∈ R D for each p ∈ R 2 , such that z p is compact in the geometric concentrated part
S p and contrast otherwise.
Specifically, we use the Soft-Nearest Neighbor Loss (Salakhutdinov and
Hinton, 2007) to formulate such:
[Equation]
where z p ∗ ∈ z is the per-pixel elastic part representation on pixel location p ∗ , ⟨· , ·⟩ is the dot product,and τ is the temperature.
Finally, we use a ConvNet, F Θ ( · ) , to model the generation of the elastic part representation, i.e., forthe reference image x ∈ R H
× W × 3 , F Θ ( x ) ∈
R H
× W × D denotes the collection of per-pixel elasticpart representations.
3.2 Part-aware Matching
From the very beginning, the depth map and normal map are randomly initialized under the uniformdistribution.
After training the part representation (described in Sec. 3.3), we build our part-awarematching module on top of the seminal work of patchmatch (Galliani et al., 2015).
In particular, weincorporate our elastic part representation and propose the following two novel improvements: 1.
Propagation: we propagate hypotheses to neighbors with our learned elastic part representation; 2.
Evaluation: we evaluate and choose the best hypotheses based on photometric consistency, featurelevel correspondence, and geometrically smoothness.
The resulting module is differentiable, and thearchitecture of part-aware patchmatch is illustrated in Fig.
Propagation.
The key idea of the part-aware propagation is to gather hypotheses from the nearbypixels of the same physical surface part.
As illustrated in Fig.
3, instead of propagating hypothesesnaively from a static set of neighbors (Galliani et al., 2015), our depth hypotheses are sampled fromthe nearby pixels which own similar features of the learned elastic part representation.
Since the depth value is unreliable in textureless and highly reflectant areas, a confidence mapc ∈ [0 , 1] H × W is generated to pick the reliable candidates from noise, which is a combination of thephotometric consistency and the geometric consistency (Schönberger et al., 2016).
Please refer tothe supplementary material for a more detailed explanation.
For each pixel p in the depth map to beupdated, K p patch hypotheses are propagated from the propagation candidates T p , which is a set ofnearby pixels q whose feature z q is close to the feature z p with high confidence c q :
[Equation]
According to the property of elastic part representation in Eq. 1, the generated depth hypotheses from
T p would stay close to the underlying surface.
Evaluation.
Beyond the photometric consistency loss, we propose two novel losses for hypothesisevaluation – Part-aware correspondence loss and Part smoothness loss.
Part-aware correspondence loss evaluates the feature similarity of the elastic part representationbetween the reference view and the warped source views.
Given in homogeneous coordinates anddepth hypothesis d , we obtain the warped source feature z
[ i ] p i ( d ) of the i-th source view via differentiablebilinear interpolation.
Therefore, given the features z [0] p , z
[ i ] p i ( d ) of the elastic part representation inthe reference and source view respectively, the part-aware correspondence at depth d p and normaldirection n p is given by:
[Equation]
where ρ ( · ) is the NCC value (Galliani et al., 2015) implemented in the traditional patchmatchalgorithm.
Part smoothness loss aims to augment the local patchmatch evaluation with a non-local piecewisesmoothness term corresponding to the same physical surface part.
Inspired by the work of Locally
Optimal Projection (LOP) (Lipman et al., 2007) for surface reconstruction from point clouds, the partsmoothness loss is formulated using an L1 median loss, which is robust to the outliers and gives apiecewise second-order surface approximation in each surface part (Lipman et al., 2007).
More specifically, the depth map d
∈ R H
× W × 1 is firstly transferred to the point set e ∈ R
H
× W × 3in the scene space given the camera matrix M : e = { e p ∗ ∈ R
| [ e ⊤ p ∗ , 1] ⊤ =
M − 1 [ p ∗⊤ , d p ∗ , 1] ⊤ } .
Then we define our part smoothness loss as the sum of the Euclidean distance to the points frompropagation candidates in T p
(Eq. 2):
[Equation]
where ω q = exp {− c p − α n ⟨ n p , n q ⟩} is a weight correlated with the confidence c q and the normalsimilarity.
Putting all the components together, as illustrated in Fig.
2, the depth map initialization consistsof K p random values within the depth range of interest.
For subsequent iterations, K p hypotheses are propagated based on the depth values of the former one.
To deliver a more diverse set ofhypotheses than just using propagation, another K r hypotheses are sampled by adding a smallrandom perturbation to the previous estimation.
In each iteration, the optimal depth and normalare chosen as: ( d opt p , n opt p ) =
argmin d ∗ p ,n ∗ p (cid:8) M s ( d ∗ p , n ∗ p | x, z ) + α g · M g ( d ∗ p , n ∗ p | z ) (cid:9) .
This function isnot directly solved using the first-order optimization mechanisms such as gradient descent, butrather approximated using a discrete sampling strategy, which has long been proposed in the fieldof stereo-matching (Schönberger et al., 2016).
Empirically, 5 iterations are sufficient to obtain anaccurate piecewise smoothness depth map.
3.3 Network Training
The elastic part representation z , as described in Sec. 3.1, is compact in the area of a smooth surfacepart.
We now describe how to train the ConvNet F Θ ( x ) using the self-supervised contrastive learningstrategy without knowing the ground-truth depth and normal information.
The core of contrastive learning is to construct a set of positive and negative samples, in whichthe representation in the positive samples stay close to each other while the negative ones are farapart.
Before training, a depth map d and a normal map n of each reference image x are propagatedand generated by the initialized ElasticMVS without the part representation, which is similar to thetraditional patchmatch-based algorithm, e.g., Gipuma (Galliani et al., 2015).
When selecting positive and negative samples during the training stage, we get rid of the noise anderror in the initial depth map using the confidence map.
For each pixel p with high confidence( c p ≥ ξ ), we construct a set of pixel candidates ˆ S p for positive samples, in which each of them isgeometrically concentrated to p with high confidence:
[Equation]
Therefore, the dense contrastive loss for self-supervised training is defined as:
[Equation]
This loss function encourages representation compactness in parts with close surface distance andseparation otherwise, i.e., the distance in the representation space naturally reflects the distance in the3D scene space.
Since this loss function only acts on pixels with above-the-threshold confidence,it prevents the initial reconstruction artifacts from ruining the contrastive learning process.
Due tothe memory limitation, it is impossible to densely sample all positive and negative samples fromthe whole image in Eq.
Therefore, we randomly select N c points from all samples in ˆ S p , using aprobability distribution inversely proportional to the distance away from p .
To obtain spatially concentrated representation, we utilize a spatial concentration loss (Yang et al. ,2020) that encourages all the pixel embeddings to become isotropically isolated:
[Equation]
where β is a constant parameter that controls the weight of the feature similarity between p and q ,which prompts the weighted average of the sampled neighboring points q to be close to the point p .
Therefore, the pixel locations far away from the sampled point will be given low weights, leadingto a spatially concentrated representation.
The concentration loss enables the spread of reasonabledistance between each representation from high-confidence regions to low ones, which enables arobust training of the representation towards noisy and incomplete depth predictions.
A more detailedanalysis is demonstrated in Table 4 and Fig. 6.
Our final loss function is a weighted sum of contrastive loss and concentration loss, defined as :
[Equation]
where γ s denotes the weight of concentration loss in the total loss function.
3.4 Implementation Details
[Table] [Table] The elastic part representation is trainedwith a Feature Pyramid Network , whichshares the same structure as in PatchmatchNet (Wang et al., 2021a).
To train thenetwork, we use the SGD (Bottou, 2010)with an initial learning rate of 1 . {+
+} × 10 − 4 .
The threshold ϵ for defining S p is set to0 .
5 in the DTU dataset and 0 .
01 in the T&T dataset.
We fix the temperature number τ = 0 .
5 and the confidence thresholdξ = 0 .
7 during training and propagation.
We sample N c = 8 points for each pixel.
During the propagation, we select K p = 8hypotheses for propagation and K r = 4for perturbation.
We choose η = 0 .
1 in Eq.
2 to find nearby pixels.
During thereconstruction stage, we set respectivelyα s = 0 .
1 and α g = 0 .
5 in Eq.
3 and 4.
For the loss functions, β = 0 .
5 in Eq.
7and γ s = 0 . 005 in Eq. All the trainingand reconstruction are conducted on a single NVIDIA GTX 3090 graphics card.
Experiment
4.1 Benchmarks
DTU dataset (Aanæs et al., 2016) is an MVS dataset consisting of different indoor objects scanned at49 camera positions under 7 different lighting conditions.
We use the original image size ( 1600 × 1200 )as input and follow the same evaluation protocol as (Aanæs et al., 2016) to compute the accuracy and the completeness of the final reconstructions.
The overall distance metric is defined as the mean ofthe two metrics.
Quantitative results with three types of reconstruction methods are shown in Table 1, includinggeometry-based, supervised and self-supervised MVS methods.
Our method outperforms all theself-supervised methods including SurRF (Zhang et al., 2021) and JDACS (Xu et al., 2021) inaccuracy and overall distance metrics, and achieves the second-best performance in completeness .
Itis remarkable that ElasticMVS is competitive compared to state-of-the-art supervised methods, e
[Table]
g.,our overall distance metric is the second best among all methods.
[Table]
Tanks and Temples (Knapitsch et al., 2017)dataset contains two benchmarks ( intermediateand advanced ) on outdoor real-world sceneswith complex geometry.
We use the originalimage size (1920 × 1080) as the input.
Weevaluate the results with F-score (Knapitschet al., 2017), the higher value of which indicatesthe better reconstruction.
We summarize thequantitative results in Table 2.
On the intermediate benchmark, ElasticMVS achieves stateof-the-art performance among all supervisedand self-supervised MVS methods.
Especially,our method outperforms all the self-supervised
MVS methods by a significant margin.
On themore challenging T&T advanced benchmark,which contains large-scale real-world sceneswith more than 300 camera views for each scene,
ElasticMVS even compares favourably with thesupervised methods.
The tremendous performance improvement in this benchmark demonstrates the effectiveness and robustness of
the proposed
ElasticMVS w.r.t complex categories and geometries.
[Table]
We also compare our reconstructions with various baselines qualitatively in Fig.
ElasticMVSproduces much more complete predictions compared with COLMAP (Schönberger et al.,
2016),while generating much less noise than existing selfsupervised methods (M3VSNet (Huang et al. , 2021)and JDACS (Xu et al., 2021)).
Besides, we achieveeven surpassing performances upon state-of-the-artsupervised methods (CasMVSNet (Gu et al.
, 2019)and PatchmatchNet (Wang et al., 2021a)).
4.2 Ablation Study
In this section, we provide detailed ablation studyand analysis of the key components of ElasticMVS, including the part-aware patchmatch and theelastic part representation.
Part-aware patchmatch.
To investigate the effects of the key components in the part-aware patchmatch during inference, we conduct a comparison with respect to the part-aware propagation andevaluation described in Sec. 3.2.
Our part-aware patchmatch is compared with three variants: 1.
Propagation without elastic Part Representation (w/o PR), which employs fixed sampled locations forpropagation similar to Gipuma (Galliani et al., 2015); 2.
Evaluation without Part-aware Correspondence (w/o PC), which only employs the photometric consistency in Eq. 3; 3.
Evaluation without Part
Smoothness loss (w/o PS) in Eq. 4, which simply employs an L1-median loss from all neighborhoods,similar to the LOP (Lipman et al., 2007) for point cloud de-noising.
As shown in Table 3, our part-aware propagation and evaluation modules both play important roles inboosting the overall performance across different datasets.
Qualitative results are shown in Fig.
Implementations without the elastic part representation (a) and the part-aware correspondence (b) fail to recover the dense surface in textureless regions.
Although applying a naive surface de-noisingand completion method (c) helps to fill in some of the holes in textureless areas, it still suffers fromlow accuracy and noisy predictions due to the lack of part awareness.
Overall, the strong geometriccoherence revealed by the learned elastic sampling strategy helps ElasticMVS gather more accurateand complete depth maps in both small complex surfaces and large flattened areas.
[Table]
Training loss of elastic part representation.
Nowwe study how different training losses described in
Sec. 3.3 affect the performance of the representationand further the depth map predictions.
Our elasticpart representation is compared with two variants,which are trained without confidence map (w/o Con.)in Eq.
6 and without spatial concentration loss (w/o
Spa.)
in Eq. 7.
We test on the DTU dataset.
We first measure thecompactness of the learned part representation asdescribed in Eq.
1 using the ground truth depth map.
We also compare the predicted depth map error using these representations, which is evaluated by the
L1 distance from the predicted depth map to the ground-truth.
As shown in Table 4, our elastic partrepresentation is the most compact one (with lowest contrastive loss).
Visualizations of the representation are demonstrated in Fig. 6, shown by a heat map of the representation distance ∥
z q − z p ∥ from a pixel location p to all other pixels q in the whole image.
confidence map, and Spa. denotes spatial concentration loss. As expected, our elastic part representation successfully recognizes the part onthe same physical surface, resulting inproper surface boundary identificationswith elastically-varying scales and shapes.
Contrastingly, the representation trainedwithout the confidence map (w/o Con.)fails to identify a complete surface part dueto the noisy training samples.
Meanwhile,the representations trained without spatialconcentration loss (w/o Spa.) suffer fromoutliers far from the surface part, demonstrating the effectiveness of spatial regularization.
4.3 Discussion
Memory.
In Table 5, we compare the GPUmemory consumption and the run-time during inference with two memory-efficientlearning-based methods.
We choose scan 9 and a fixed reference view downsampled to the scaleof 640 × 800 in the DTU dataset in this study.
As shown in Table 5, ElasticMVS is more memoryefficient compared with two learning-based MVS backbones.
Notice that the memory and running [Table] [Table] time are adaptively changed with the hypothesis number and the iteration number, since propagationwith more hypotheses will converge with fewer iterations, but take a larger memory consumption.
Overall, the experiment demonstrates the superiority memory efficiency of ElasticMVS.
Impact of parameters. To better understand the effect of different parameters, we first evaluate thereconstruction quality w.r.t different thresholds. As shown in Table 6, with lower ϵ , the reconstructionaccuracy grows higher while the completeness becomes lower. The reason is that ϵ (point-to-planedistance) controls the compactness of the surface.
Smaller ϵ during inference leads to a more strictsurface boundary with smaller regions.
On the other hand, with a higher confidence threshold ξ , the accuracy and the overall performance ofthe prediction become higher.
With a loose threshold ( ξ = 0 . 5 ), the performance drops dramatically.
Therefore, it is better to set the confidence threshold at a high level.
Limitation and Future work.
It takes a long time in ElasticMVS to calculate the patch similarity foreach pixel, since the total computational complexity to evaluate the photo consistency from M sourceimages of size H × W with K hypotheses is O ( M × H × W × K × l 2 ) , where l is the patch size.
Empirically, l = 11 , which is impossible to give a forward and backward pass from a single graphicscard at once.
As shown in Table 5, the running time of ElasticMVS is not competitive compared withthe cost-volume-based methods.
One solution is to replace the l × l patch with the learned featurein Patchmatch.
However, the accuracy of the learned photometric consistency in self-supervisedstereopsis is still far from the traditional one.
In future work, we plan to design a lightweight networkarchitecture to represent not only the part correlation but also the pixel-wise wrapped image patches,jointly learned from a self-supervised learning-based strategy, which can meet the demand for morechallenging tasks such as real-time MVS and dynamic MVS.
Conclusion
In this paper, we present a novel elastic part representation that encodes physically connected partsegmentations with elastically-varying scales, shapes and boundaries.
We build the correlationbetween the embedding space and the scene space to represent the surface connectedness andboundaries behind the image.
A self-supervised MVS framework, ElasticMVS, is proposed tolearn the representation and estimate per-view depth following a part-aware patchmatch-basedalgorithm.
Benefiting from the inherent geometric correlations learned from the huge amount ofimages, ElasticMVS delivers state-of-the-art performance in both reconstruction completeness andaccuracy, and extraordinary generalization ability in the challenging large-scale scenes.