ElasticMVS: Learning elastic part representation for self-supervised multi-view stereopsis
Abstract
Self-supervised multi-view stereopsis (MVS) attracts increasing attention for learning dense surface predictions from only a set of images without onerous groundtruth 3D training data for supervision.
However, existing methods highly rely on the local photometric consistency, which fail to identify accurately dense correspondence in broad textureless or reflectant areas.
In this paper, we show that geometric proximity such as surface connectedness and occlusion boundaries implicitly inferred from images could serve as reliable guidance for pixel-wise multi-view correspondences.
With this insight, we present a novel elastic part representation, which encodes physically-connected part segmentations with elastically-varying scales, shapes and boundaries.
Meanwhile, a self-supervised MVS framework namely ElasticMVS is proposed to learn the representation and estimate per-view depth following a part-aware propagation and evaluation scheme.
Specifically, the pixel-wise part representation is trained by a contrastive learning-based strategy, which increases the representation compactness in geometrically concentrated areas and contrasts otherwise.
ElasticMVS iteratively optimizes a part-level consistency loss and a surface smoothness loss, based on a set of depth hypotheses propagated from the geometrically concentrated parts.
Extensive evaluations convey the superiority of ElasticMVS in the reconstruction completeness and accuracy, as well as the efficiency and scalability.
Particularly, for the challenging large-scale reconstruction benchmark, ElasticMVS demonstrates significant performance gain over both the supervised and self-supervised approaches.
Code is avaliable at https://thu-luvision.github.io.
Introduction
Given a set of images with known camera parameters, multi-view stereopsis (MVS) aims to reconstruct the dense and accurate geometry of the scene.
Existing learning-based approaches [47], typically being cast as an end-to-end depth regression task, outperform the traditional geometry-based ones [34].
However, their generalization ability in open environments is severely hindered by the availability of the onerous laser-scanned 3D training data.
Self-supervised MVS [24, 11] lifts such limitation by leveraging the multi-view photometric consistency in place of the supervisory signals.
Unfortunately, the pixel-level photometric regularization is susceptible to textureless patterns and illumination variations, leading to incomplete and inaccurate reconstructions, especially in outdoor environments.
In this paper, we tackle the above challenges with a novel image parsing paradigm.
Instead of treating images as projections of 3D scenes, we show that the inherent geometric correlations, such as surface connectedness, smoothness and boundaries, could be inferred implicitly from images and serve as a reliable guidance for pixel-wise multi-view correspondence estimation.
The mainstream solutions of feature-based regularizations introduce either hand-crafted structures such as lines and planes from superpixels [32, 4, 30], or semantic features pre-trained from limited geometricunrelated labels [43, 17].
Different from those, we aim to build the correlation between the latent space of the representation and the scene space of the surface in a self-supervised way.
With this insight, we present a novel elastic part representation which encodes physically-connected part segmentations with elastically-varying scales, shapes and boundaries.
Unlike the traditional segmentation networks with a fixed number of labels, our pixel-wise part representation directly reflects the surface correlation, i.e., the more similar the two representations are, the higher probability they belong to the same physical surface part.
Meanwhile, a self-supervised MVS framework namely
ElasticMVS is proposed to learn the representation and estimate per-view depth iteratively following a part-aware propagation and evaluation scheme.
We believe that this framework provides a neat solution to bridge the gap between the multi-view stereopsis and the point cloud reconstruction [3], which has the potential to be effectively exploited by many other MVS methods.
Benefiting from the elastic part representation, the inherent geometric correlations are effectively exploited by a patchmatch-based propagation strategy [13].
Specifically, in each iteration, depth is estimated by selecting the best depth hypotheses adaptively propagated only from the geometrically concentrated part.
Different from other adaptive propagation strategy [39, 44] based only on pure image features without explicit geometric constraints, our part-aware propagation scheme directly relies on physically plausible indicators.
Besides, the pixel-level photometric consistency loss is enhanced by the constraints of feature-level correspondence and non-local piecewise surface smoothness, which provide part-aware matching cues to guide a more robust depth prediction in textureless and reflectant areas.
To train the elastic part representation, a contrastive learning-based strategy is implemented by increasing the representation compactness in geometrically concentrated areas and contrasts otherwise.
Without the ground-truth 3D information, these geometrically concentrated areas are identified by the depth map generated from the initialized ElasticMVS without the part representation, which owns the same workflow as the traditional geometry-based stereopsis [34].
A noise-robust training loss is implemented to get rid of undesirable identifications.
Experiments on ‘DTU’ and ‘Tanks and Temples (T&T)’
datasets demonstrate that the proposed
ElasticMVS significantly outperforms all state-of-the-art self-supervised learning-based methods.
Remarkably, in the challenging large-scale reconstruction benchmark (T&T advanced), ElasticMVS shows dramatic performance improvements w.r.t.
both the supervised and self-supervised approaches.
Besides, the systematic evaluations indicate that our elastic part representation is superior in modeling elastically-varied parts, while being robust and lightweight in training and inference.
Related work 2.1 Multi-view Stereopsis (MVS)
The major works in multi-view stereopsis focus on the view-wise depth map estimation [36, 12] and then fuse together to form the point cloud[50, 14].
Among these methods, patchmatch-based depth optimization methods[13, 34, 44], initialized from the randomly sampled depth hypotheses, show great run-time and memory efficiency in depth map estimation.
Some learning-based methods [39, 26] introduce an iterative coarse-to-fine optimization in an end-to-end trainable architecture to improve each of the core steps in patchmatch.
However, these algorithms suffer from incomplete and noisy depth predictions in textureless and reflectant areas.
Several previous works in MVS solved the problem by introducing pre-defined global structural priors such as planes to resolve the ambiguity.
For instance, TAPAMVS [32] segments the whole image into a bunch of superpixels and assumes textureless areas piecewise planar.
RANSAC is applied to estimate its corresponding plane parameters.
Beyond that, a bunch of works in point cloud reconstruction deal with reconstruction artifacts with outlier, noisy or missing data.
For instance, implicit representation [22] is used to provide global surface smoothness prior given noisy data, and exterior visibility [21] is used to detect and eliminate redundant data.
However, neither the handcrafted plane structure nor the complex smoothness prior can provide sufficiently diverse and accurate surface part modeling for real-world scenarios.
Currently, learning methods based on 3D cost volume regularization demonstrate state-of-the-art results in several benchmarks.
These methods optimizes the 3D geometry by 3D CNN, unprojecting from 2D images or features into the 3D volume [18, 19, 23, 47, 20, 48, 49, 7, 15, 10, 28, 52].
However, these supervised methods crucially rely on ground-truth 3D data, which are hard to generalize in wild open environments.
To realize self-supervised MVS without the onerous ground-truth 3D training data, [11, 16, 24, 51, 43, 29, 31] replace the supervision signal with the unsupervised rendering loss or cross-view feature consistency loss.
However, reprojection under multiple views is highly sensitive to environmental illumination, which is hard to generalize in highly variable scenarios.
2.2 Part Representation
Recently, learning representations from the image in a self-supervised training strategy has gained growing popularity in 2D vision tasks, which use contrastive learning [42, 8, 9, 6] to deal with the lack of ground-truth labels.
They use InfoNCE [38] to generate a unique embedding for each picture in a batch.
[40] further leverages a dense projection head to generalize such embedding from image-level to patch-level, which is more suitable for performing dense prediction tasks such as semantic segmentations.
[17, 41] assigns the equivariance constraint and the semantic consistency regularization on the learned part response representation [53] to predict part segmentations that are consistent across different instances or camera poses.
[43] incorporates learned semantic features into multi-view stereopsis pipeline to boost the reconstruction ability.
However, previous works only concentrate on generating semantic features and using it forward to perform down-stream tasks.
Instead, we utilize the rough depth map as supervision to the semantic learning process, which forms a natural integration of photometric and geometric information to provide elastic part representations.
Method
We estimate the depth maps of a given reference image x and the corresponding M unstructured source images with known camera calibrations.
The final point cloud reconstruction is obtained by fusing all of them.
Since the geometry information gained from any photometric loss contains missing data and artifacts in some areas of the image (especially in textureless regions), a heuristic geometry regularization for depth prediction is unavailable in these regions.
Therefore, the key innovation in our framework is a novel elastic part representation , which encodes sufficient geometric details to guide a piecewise-smooth depth map prediction.
We first formally define our part-aware representation in
Sec. 3.1.
Based upon the learned representation, we introduce two improvements on depth hypotheses propagation and evaluation for part-aware patchmatch in Sec. 3.2.
We present the self-supervised training scheme in Sec.
3.3 to robustly train the representation with noisy initialization.
3.1 Elastic Part Representation
Suppose that ( ˜ d, ˜ n ) ∈
R H
× W × 4 are the underlying depth and normal map of the image x , we first define the geometry-aware part segmentation S p related to each pixel location p ∈ R 2 , which is geometrically concentrated inside: S p ≜ { q ∈ R 2 | Γ { ( ˜ d p , ˜ n p ); ( ˜ d q , ˜ n q ) } ≤ ϵ } , where Γ { ( ˜ d p , ˜ n p ); ( ˜ d q , ˜ n q ) } ≜ | ([ q, ˜ d q ] − [ p, ˜ d p ]) · ˜ n p | is the 3D point-to-plane distance [35] between two pixels.
Equipped with the part segmentation S p for each pixel p , we propose to learn a per-pixel elastic part representation z p ∈ R D for each p ∈ R 2 , such that z p is compact in the geometric concentrated part
S p and contrast otherwise.
Specifically, we use the Soft-Nearest Neighbor Loss [33] to formulate such:
[Equation]
where z p ∗ ∈ z is the per-pixel elastic part representation on pixel location p ∗ , ⟨· , ·⟩ is the dot product, and τ is the temperature.
Finally, we use a ConvNet, F Θ ( · ) , to model the generation of the elastic part representation, i.e., for the reference image x ∈
R H
× W × 3 , F Θ ( x ) ∈
R H
× W × D
denotes the collection of per-
pixel elastic
part representations.
3.2 Part-aware Matching
After training the part representation (described in Sec. 3.3), we build our part-aware matching module on top of the seminal work of patchmatch [13].
That is, the final depth prediction is iteratively refined from the beginning of the random initialization.
In particular, we incorporate our elastic part representation and propose the following two novel improvements: 1.
Propagation: we propagate hypotheses to neighbors with our learned elastic part representation; 2.
Evaluation: we evaluate and choose the best hypotheses based on photometric consistency, feature-level correspondence, and geometrically smoothness.
The resulting module is differentiable, and the architecture of part-aware patchmatch is illustrated in Fig.
Propagation.
The key idea of the part-aware propagation is to gather hypotheses from the nearby pixels of the same physical surface part.
As illustrated in Fig.
3, instead of propagating hypotheses naively from a static set of neighbors [13], our depth hypotheses are sampled from the nearby pixels which own similar features of the learned elastic part representation.
Since the depth value is unreliable in textureless and highly reflectant areas, a confidence map c ∈ [0 , 1] H × W is generated to pick the reliable candidates from noise, which is a combination of the photometric consistency and the geometric consistency [34].
Please refer to the supplementary material for a more detailed explanation.
For each pixel p in the depth map to be updated, K p patch hypotheses are propagated from the propagation candidates T p , which is a set of nearby pixels q whose feature z q is close to the feature z p with high confidence c q :
[Equation]
According to the property of elastic part representation in Eq. 1, the generated depth hypotheses from
T p would stay close to the underlying surface.
Evaluation.
Beyond the photometric consistency loss, we propose two novel losses for hypothesis evaluation – Part-aware correspondence loss and Part smoothness loss.
Part-aware correspondence loss evaluates the feature similarity of the elastic part representation between the reference view and the warped source views.
Given in homogeneous coordinates and depth hypothesis d , we obtain the warped source feature z
[ i ] p i ( d ) of the i-th source view via differentiable bilinear interpolation.
Therefore, given the features z [0] p , z
[ i ] p i ( d ) of the elastic part representation in the reference and source view respectively, the part-aware correspondence at depth d p and normal direction n p is given by:
[Equation]
where ρ ( · ) is the bilateral weighted normalized cross correlation [34] implemented in the traditional patchmatch algorithm.
Part smoothness loss aims to augment the local patchmatch evaluation with a non-local piecewise smoothness term corresponding to the same physical surface part.
Inspired by the work of Locally
Optimal Projection (LOP) [27] for surface reconstruction from point clouds, the part smoothness loss is formulated using an L1 median loss, which is robust to the outliers and gives a piecewise second-order surface approximation in each surface part [27].
More specifically, the depth map d
∈ R H
× W × 1 is firstly transferred to the point set
e ∈ R
H
× W ×in the scene space given the camera matrix
M : e = { e p ∗ ∈ R 3 | [ e ⊤ p ∗ , 1] ⊤ =
M − 1 [ p ∗⊤ , d p ∗ , 1] ⊤ } .
Then we define our part smoothness loss as the sum of the Euclidean distance to the points from propagation candidates in T p (Eq. 2):
[Equation]
where ω q = exp {− c p − α n ⟨ n p , n q ⟩} is a weight correlated with the confidence
c q and the normal similarity.
Putting all the components together, as illustrated in Fig.
2, the depth map initialization consists of K p random values within the depth range of interest.
For subsequent iterations, K p hypotheses are propagated based on the depth values of the former one.
To deliver a more diverse set of hypotheses than just using propagation, another K r hypotheses are sampled by adding a small random perturbation to the previous estimation.
In each iteration, the optimal depth and normal are chosen as: ( d opt p , n opt p ) =
argmin d ∗ p ,n
[Equation]
p
[Equation]
+
[Equation]
g
[Equation]
g
[Equation]
d
[Equation]
p
[Equation]
n
[Equation]
p
[Equation]
z ) (cid:9) .
This function is not directly solved using the first-order optimization mechanisms such as gradient descent, but rather approximated using a discrete sampling strategy, which has long been proposed in the field of stereomatching [34].
Empirically, 5 iterations are sufficient to obtain an accurate piecewise smoothness depth map.
3.3 Network Training
The elastic part representation z , as described in Sec. 3.1, is compact in the area of a smooth surface part.
We now describe how to train the ConvNet F Θ ( x ) using the self-supervised contrastive learning strategy without knowing the ground-truth depth and normal information.
The core of contrastive learning is to construct a set of positive and negative samples, in which the representation in the positive samples stay close to each other while the negative ones are far apart.
Before training, a depth map d and a normal map n of each reference image x are propagated and generated by the initialized ElasticMVS without the part representation, which is similar to the traditional patchmatch-based algorithm, e.g., Gipuma [13].
When selecting positive and negative samples during the training stage, we get rid of the noise and error in the initial depth map using the confidence map.
For each pixel p with high confidence ( c p ≥ ξ ), we construct a set of pixel candidates ˆ S p for positive samples, in which each of them is geometrically concentrated to p with high confidence:
[Equation]
Therefore, the dense contrastive loss for self-supervised training is defined as:
[Equation]
This loss function encourages representation compactness in parts with close surface distance and separation otherwise, i.e., the distance in the representation space naturally reflects the distance in the 3D scene space.
Since this loss function only acts on pixels with above-the-threshold confidence, it prevents the initial reconstruction artifacts from ruining the contrastive learning process.
Due to the memory limitation, it is impossible to densely sample all positive and negative samples from the whole image in Eq.
Therefore, we randomly select N c points from all samples in ˆ S p , using a probability distribution inversely proportional to the distance away from p .
To obtain spatially concentrated representation, we utilize a spatial concentration loss [46] that encourages all the pixel embeddings to become isotropically isolated:
[Equation]
where β is a constant parameter that controls the weight of the feature similarity between p and q , which prompts the weighted average of the sampled neighboring points q to be close to the point p .
Therefore, the pixel locations far away from the sampled point will be given low weights, leading to a spatially concentrated representation.
The concentration loss enables the spread of reasonable distance between each representation from high-confidence regions to low ones, which enables a robust training of the representation towards noisy and incomplete depth predictions.
A more detailed analysis is demonstrated in Table 4 and Fig.
Our final loss function is a weighted sum of contrastive loss and concentration loss, defined as :
[Equation]
where γ s denotes the weight of concentration loss in the total loss function.
3.4 Implementation Details
The elastic part representation is trained with a Feature Pyramid Network , which shares the same structure as in PatchmatchNet [39].
To train the network, we use the SGD [5] with an initial learning rate of 1 .
× 10 − 4 .
The threshold ϵ for defining S p is set to 0 .
5 in the DTU dataset and 0 .
01 in the
T&T dataset.
We fix the temperature number τ = 0 .
5 and the confidence threshold ξ = 0 .
7 during training and propagation.
We sample N c = 8 points for each pixel.
During the propagation, we select
K p = 8 hypotheses for propagation and K r = 4 for perturbation.
We choose η = 0 .
1 in Eq.
2 to find nearby pixels.
During the reconstruction stage, we set respectively α s = 0 .
1 and α g = 0 .
5 in
Eq. 3 and 4.
For the loss functions, β = 0 .
5 in Eq.
7 and γ s = 0 .
005 in Eq.
All the training and reconstruction are conducted on a single NVIDIA GTX 3090 graphics card.
Experiment 4.1 Benchmarks
[Table]
DTU dataset [1] is an MVS dataset consisting of different indoor objects scanned at 49 camera positions under 7 different lighting conditions.
We use the original image size ( 1600 × 1200 ) as input and follow the same evaluation protocol as [1] to compute the accuracy and the completeness of the final reconstructions.
The overall distance metric is defined as the mean of the two metrics.
Quantitative results with three types of reconstruction methods are shown in Table 1, including geometry-based, supervised and selfsupervised MVS methods.
Our method outperforms all the self-supervised methods including SurRF [51] and JDACS [43] in accuracy and overall distance metrics, and achieves the second-best performance in completeness .
It is remarkable that ElasticMVS is competitive compared to state-of-the-art supervised methods, e.g., our overall distance metric is the second best among all methods.
[Table]
Tanks and Temples [25] dataset contains two benchmarks ( intermediate and advanced ) on outdoor real-world scenes with complex geometry.
We use the original image size (1920 × 1080) as the input.
We evaluate the results with
F-score [25], the higher value of which indicates the better reconstruction.
We summarize the quantitative results in Table 2.
On the intermediate benchmark, ElasticMVS achieves stateof-the-art performance among all supervised and self-supervised MVS methods.
Especially, our method outperforms all the self-supervised
MVS methods by a significant margin.
On the more challenging T&T advanced benchmark, which contains large-scale real-world scenes with more than 300 camera views for each scene,
ElasticMVS even compares favourably with the supervised methods.
The tremendous performance improvement in this benchmark demonstrates the effectiveness and robustness of the proposed ElasticMVS w.r.t complex categories and geometries.
We also compare our reconstructions with various baselines qualitatively in Fig.
ElasticMVS produces much more complete predictions compared with COLMAP [34], while generating much less noise than existing self-supervised methods (M3VSNet [16] and JDACS [43]).
Besides, we achieve even surpassing performances upon state-of-the-art supervised methods (CasMVSNet [15] and PatchmatchNet [39]).
4.2 Ablation Study
In this section, we provide detailed ablation study and analysis of the key components of ElasticMVS, including the part-aware patchmatch and the elastic part representation.
Part-aware patchmatch.
To investigate the effects of the key components in the part-aware patchmatch during inference, we conduct a comparison with respect to the part-aware propagation and evaluation described in Sec. 3.2.
Our part-aware patchmatch is compared with three variants: 1.
Propagation without elastic Part Representation (w/o PR), which employs fixed sampled locations for propagation similar to Gipuma [13]; 2.
Evaluation without Partaware Correspondence (w/o PC), which only employs the photometric consistency in Eq. 3; 3.
Evaluation without Part Smoothness loss (w/o PS) in Eq. 4, which simply employs an
L1-median loss from all neighborhoods, similar to the LOP [27] for point cloud de-noising.
[Table]
As shown in Table 3, our part-aware propagation and evaluation modules both play important roles in boosting the overall performance across different datasets.
Qualitative results are shown in Fig.
Implementations without the elastic part representation (a) and the part-aware correspondence (b) fail to recover the dense surface in textureless regions.
Although applying a naive surface de-noising and completion method (c) helps to fill in some of the holes in textureless areas, it still suffers from low accuracy and noisy predictions due to the lack of part awareness.
Overall, the strong geometric coherence revealed by the learned elastic sampling strategy helps
ElasticMVS gather more accurate and complete depth maps in both small complex surfaces and large flattened areas.
Training loss of elastic part representation.
Now we study how different training losses described in Sec.
3.3 affect the performance of the representation and further the depth map predictions.
Our elastic part representation is compared with two variants, which are trained without confidence map (w/o Con.) in Eq.
6 and without spatial concentration loss (w/o Spa.)
in Eq. 7.
[Table]
We test on the DTU dataset.
We first measure the compactness of the learned part representation as described in Eq.
1 using the ground truth depth map.
We also compare the predicted depth map error using these representations, which is evaluated by the L distance from the predicted depth map to the groundtruth.
As shown in Table 4, our elastic part representation is the most compact one (with lowest contrastive loss).
Visualizations of the representation are demonstrated in Fig. 6, shown by a heat map of the representation distance ∥
z q − z p ∥ from a pixel location p to all other pixels q in the whole image.
As expected, our elastic part representation successfully recognizes the part on the same physical surface, resulting in proper surface boundary identifications with elastically-varying scales and shapes.
Contrastingly, the representation trained without the confidence map (w/o Con.) fails to identify a complete surface part due to the noisy training samples.
Meanwhile, the representations trained without spatial concentration loss (w/o Spa.) suffer from outliers far from the surface part, demonstrating the effectiveness of spatial regularization.
4.3 Discussion
Memory.
In Table 5, we compare the GPU memory consumption and the run-time during inference with two memory-efficient learning-based methods.
We choose scan 9 and a fixed reference view downsampled to the scale of 640 × 800 in the DTU dataset in this study.
As shown in Table 5,
ElasticMVS is more memory-efficient compared with two learning-based MVS backbones.
Notice that the memory and running time are adaptively changed with the hypothesis number and the iteration number, since propagation with more hypotheses will converge with fewer iterations, but take a larger memory consumption.
Overall, the experiment demonstrates the superiority memory efficiency of ElasticMVS.
Impact of parameters.
To better understand the effect of different parameters, we first evaluate the reconstruction quality w.r.t different thresholds.
As shown in Table 6, with lower ϵ , the reconstruction accuracy grows higher while the completeness becomes lower.
The reason is that ϵ (point-to-plane
[Table] [Table]
distance) controls the compactness of the surface.
Smaller ϵ during inference leads to a more strict surface boundary with smaller regions.
On the other hand
[Table]
with a higher confidence threshold ξ
[Table]
the accuracy and the overall performance of the prediction become higher.
With a loose threshold ( ξ = 0 . 5 ), the performance drops dramatically.
Therefore, it is better to set the confidence threshold at a high level.
Limitation and Future work.
It takes a long time in ElasticMVS to calculate the patch similarity for each pixel
[Table]
since the total computational complexity to evaluate the photo consistency from M source images of size H × W with K hypotheses is O ( M × H × W × K × l 2 ) , where l is the patch size.
Empirically, l = 11 , which is impossible to give a forward and backward pass from a single graphics card at once.
As shown in Table 5, the running time of ElasticMVS is not competitive compared with the cost-volume-based methods.
One solution is to replace the l × l patch with the learned feature in Patchmatch.
However, the accuracy of the learned photometric consistency in self-supervised stereopsis is still far from the traditional one.
In future work, we plan to design a lightweight network architecture to represent not only the part correlation but also the pixel-wise wrapped image patches, jointly learned from a self-supervised learning-based strategy, which can meet the demand for more challenging tasks such as real-time MVS and dynamic MVS.
Conclusion
In this paper, we present a novel elastic part representation that encodes physically connected part segmentations with elastically-varying scales, shapes and boundaries.
We build the correlation between the embedding space and the scene space to represent the surface connectedness and boundaries behind the image.
A self-supervised MVS framework, ElasticMVS, is proposed to learn the representation and estimate per-view depth following a part-aware patchmatch-based algorithm.
Benefiting from the inherent geometric correlations learned from the huge amount of images, ElasticMVS delivers state-of-the-art performance in both reconstruction completeness and accuracy, and extraordinary generalization ability in the challenging large-scale scenes.
Acknowledgements
This work is supported in part by Natural Science Foundation of China (NSFC) under contract No.
62125106, 61860206003, 62088102 and 62171256, in part by Ministry of Science and Technology of China under contract No. 2021ZD0109901 and in part by Shenzhen Key Laboratory of next generation interactive media innovative technology, Funding No: ZDSYS20210623092001004.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Sec. 4.3.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See in Sec.
3.4 and in the supplemental material.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See in the supplemental material.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See in Sec.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Appendix A Additional results A.1 Benchmarks
A detailed qualitative and quantitative comparison of state-of-the-art MVS methods on all scans in
Tanks and Temples ( T&T ) dataset, as shown in Table 7, Table 8 and Fig.7.
ElasticMVS achieves state-of-the-art performance among all supervised and self-supervised MVS methods and the tremendous performance improvement in T&T advanced benchmark, demonstrating the effectiveness and robustness of the proposed ElasticMVS to complex categories and geometries.
The qualitative comparisons of all scenes in the T&T dataset also demonstrate that ElasticMVS precisely reconstructs the scenes with higher completeness and less noise.
[Table] [Table]
[Table] [Table]
B Elastic part representation
B.
1 Training loss
Visualizations of the elastic part representation representation compared with variants trained without confidence map (w/o Con.) and spatial concentration loss (w/o Spa.), which are shown in Fig 8 and
Fig 9.
Our elastic part representation successfully recognizes the part on the same physical surface with elastically-varying scales, shapes and boundaries.
B.2 Representation Comparison
We compare our elastic part representation with part representations in SLIC [2] and JDACS [43].
Since SLIC and JDACS both represent parts using labels that are different from our embedding-based representation, we highlight those pixels sharing the same label with the reference pixel denoted by the black "+".
As shown in Fig 10, Our elastic part representations are more scalable in size while also better preserving object boundaries.
C ElasticMVS C.1 Propagation
To better understand the effect of part-aware propagation we visualize the sampled candidates for propagation (blue points) during 3 iterations of patchmatch given six pixels (red points), which are shown in Fig 11.
As expected, the receptive field for hypotheses propagation is elastically changed according to the the geometric shape and the texture pattern of the object.
C. Number of iterations
The quantitative results with different number of hypothesis ( K p , K r ) under each iteration described in Sec.
3.2 are summarized in Fig.
Obviously, the error converges after only 4 iterations of patchmatch using different number of hypotheses.
Besides, using more hypotheses leads to faster convergence.
Apart from that, our part-aware patchmatch is robust to the diverse initialization and local perturbation.
C.3 Confidence map
The depth map predicted in each iteration is not always reliable, especially in textureless and highreflectant areas.
Therefore, we integrate multi-view geometric consistency constraints into the photometric consistency to increase the reliability and the accuracy of training and propagation.
The visualization of each part is demonstrated in Fig.
We diminish reconstruction artifacts by firstly using a bilaterally weighted adaption of normalized cross correlation to measure the photometric consistency c pho p between a reference view with the corresponding source views.
After that, the geometric consistency c geo p is computed as the forward-backward reprojection error between multiple views [34].
Besides, we compute surface smoothness c smo p [35] to find smooth surface parts, which is the angle between surface normals of adjacent image points.
This term identifies sudden changes of the surface normal direction at object edges [37], which is robust w.r.t.
noisy normal vectors.
As shown in Fig. 13, for each reference image, the photometric consistency c pho p (b), geometric consistency c geo p (c) and surface smoothness c smo p (d) for each pixel p is separated computed.
The final confidence c p (e) for training and propagation is the combination of these three factors: c p =
c pho p
[Equation]
p
[Equation]