Paper: MVSFormer++: Revealing the Devil in Transformer's Details for Multi-View Stereo (ICLR-2024)

Reviewer comment (Novelty):
The paper proposed a bunch of small components/tricks over the previous MVSFormer. I acknowledge that these tricks might be useful, however, I would feel like each of them is a bit incremental and the whole story is not that interconnected. For the conference paper, I prefer a neat idea/key component that can bring strong improvements. The paper looks more like an extended journal version paper of the previous one.

Author response:
Thanks for this comment. We would like to clarify that our contributions are not merely incremental. The decision to build upon MVSFormer is strategic, as it serves as a simple yet highly effective baseline with strong scalability and significant potential. With the development of pre-trained ViTs and the widespread adoption of transformers in feature-matching tasks, as highlighted in the second paragraph of the introduction, exploring the reasonable usage of transformers in MVS becomes very important. MVSFormer++ systematically addressed three key challenges that hinder effective transformer-based learning in MVS (already previously outlined in the third paragraph of the introduction):

1) *Tailored attention mechanisms for different MVS modules.* We provide insight indicating that linear attention is well-suited for feature encoding with superior robustness in terms of sequential length, while cost volume regularization benefits more from spatial relations, favoring vanilla and window-based attention mechanisms.

2) *Incorporating cross-view information into Pre-trained ViTs.* Our Side View Attention (SVA) efficiently learns cross-view information through frozen DINOv2.

3) *Enhancing Transformer’s Length Extrapolation Capability in MVS.* We effectively address the length extrapolation issue through normalized 2D-PE, Frustoconical Positional Encoding (FPE), and Adaptive Attention Scaling (AAS) as verified in Table12.

Moreover, our neat and simple combination of Cost Volume Transformer (CVT) enhanced with FPE and AAS achieves substantial improvements based on not only MVSFormer but also CasMVSNet as shown in Table13 of Appendix A.5.
We anticipate that our work could benefit the community by bridging the gap between MVS learning and reliable transformer utilization.

Reviewer comment (Evaluation):
ETH3D evaluation: the proposed method does not perform well on ETH3D even compared with other learning-based approach (e.g., Vis-MVSNet, EPP-MVSNet). Could the authors explain potential causes?

Author response:
Thanks for this point. We need to clarify that our ETH3D results are all achieved with the same threshold (0.5) of depth confidence filter and default settings of dynamic point cloud fusion (DPCD)[1] without any cherry-pick hyper-parameter adjusting. This approach ensures a fair and unbiased evaluation. Though improvements may not be very significant, they are consistent and indicative of the robustness and competitiveness of MVSFormer++.  We have claimed these points in the paper revision. For the results of Vis-MVSNet and EPP-MVSNet, unfortunately, we could not find their detailed hyper-parameter settings for ETH3D to re-implement their results. Since our goal is to propose a generalized and powerful MVS method, we focus on the intrinsical depth estimation quality rather than sophisticated post-precessing tricks. Our codes and evaluation scripts will be fully open-released if accepted.

[1] Yan J, Wei Z, Yi H, et al. Dense hybrid recurrent multi-view stereo net with dynamic consistency checking. ECCV2020.

Reviewer comment (Experiments):
These is no a limitation section. I would like know the scalability of the proposed method, will the memory cost dramatically increased compared with CNN based approaches (e.g., CasMVSNet) when the image size/depth sample number increase?

Author response:
Thanks for this insightful comment. We have investigated the scalability in both image and depth sizes and compared MVSFormer++ to CasMVSNet in Table11. Benefiting from an efficient attention implementation[2] and a more reasonable 4-stage depth hypothesis setting, MVSFormer++ demonstrates robust scalability for dense and precise depth estimation. The development of the highly optimized attention mechanism also improves the significance of our work in advancing the effective utilization of transformers in MVS. Moreover, our CVT enhanced by FPE and AAS could also be well extended into large-scale images, which overcomes the key challenge of previous transformer-based manners.

[2] Dao T. Flashattention-2: Faster attention with better parallelism and work partitioning[J]. arXiv preprint arXiv:2307.08691, 2023.

Reviewer comment (Writing):
The paper lacks detailed explanations of specific design choices, such as the rationale behind selecting DINOv2 as the base model. Additionally, the utilization of different levels of DINOv2 features is not clearly elucidated. It is recommended to include these details to enhance the manuscript's clarity and independence.

Author response:
**1.More elucidation about the selection of DINOv2 and the utilization of different levels of DINOv2.**

Thanks for your insightful comments.  We apologize for the insufficient discussion about DINOv2. We have provided more discussions about the usage of DINOv2 and its feature layers in Sec3 Preliminary and Appendix A.1 in our paper revision.
Our choice of DINOv2[1] as the backbone of MVSFormer++ is attributed to its robust zero-shot feature-matching capability and impressive performance in dense vision downstream tasks. Compared to other ViTs, DINOv2 was trained on large-scale curated data, unifying both image-level (contrastive learning) and patch-level (masked image prediction) objectives, as well as high-resolution adaption. Hence DINOv2 enjoys remarkably stable feature correlation across different domains and better performance in dense vision benchmarks (segmentation and monocular depth estimation). Our experiments in Table10 (listed below) show the efficacy of DINOv2 in MVS.

| ViT backbone | Frozen backbone | Accuracy$\downarrow$ | Completeness$\downarrow$ | Overall$\downarrow$ |
|--------------|-----------------|----------------------|--------------------------|------------------------------------|
| DINO-small   | $\checkmark$    | 0.327                | 0.265                    | 0.296               |
| DINO-base    | $\checkmark$    | 0.334                | 0.268                    | 0.301               |
| Twins-small  | $\times$        | 0.327                | 0.251                    | 0.289               |
| Twins-base   | $\times$        | 0.326                | 0.252                    | 0.289               |
| DINOv2-base  | $\checkmark$    | **0.3198**     | 0.2549                   | 0.2875              |
| DINOv2-base  | LoRA (rank=16)  | 0.3239               | **0.2501**          | **0.2870**     |

For the selection of DINOv2 layers, we have carefully analyzed the feature characteristics in Figure6 and Table6 of the Appendix. Here we provide more detailed discussions as follows. We show the WTA depth estimations of DINOv2 in Figure6(b), which indicate middle layers are more useful to MVS. So we empirically compare the layer ablations in Table6 to verify our claims. Generally, our final combination (3,7,11) is a good selection. Though the 5th layer shows better zero-shot WTA feature correlations in Figure6(b), the gap between (3,5,11) and (3,7,11) is not obvious. As shown in Figure6(a), layers from 8 to 10 are unstable, which would cause inferior results to the 4-layer setting. More details are discussed in Appendix A.1.

We should clarify that the usage of DINOv2 and other ViT backbones are not the main contributions of this paper. Our improvements are not solely dependent on the increased capacity of pre-trained ViTs. We will explain this in the next question.

[1] Oquab M, Darcet T, Moutakanni T, et al. Dinov2: Learning robust visual features without supervision[J]. arXiv preprint arXiv:2304.07193, 2023.

Reviewer comment (Evaluation):
While the incorporation of DINOv2 in the feature extraction stage significantly enhances performance, it is crucial to clarify that this improvement is not solely due to the increase in the number of parameters.

Author response:
**2.The improvement is subtle when compared to the usage of DINOv2.**

Thanks for this point.  Although MVSFormer+DINOv2 is a strong baseline (Table10), we need to clarify that our work is orthogonal to the DINOv2. DINOv2 just improved the overall distance from 0.289 to 0.2875 with less computation (frozen backbone compared with Twins), while we further improved the overall distance from 0.2875 to 0.2805 with our novel designs of SVA, CVT, and image scaling adaption.

Reviewer comment (Experiments):
The improvement in point cloud evaluation metrics by the proposed module during ablation experiments appears subtle. To bolster the paper's experimental support, it is recommended to validate the proposed module's effectiveness by integrating it into baseline methods and conducting a comparative analysis. This would provide a clearer understanding of the module's actual contribution.

Author response:
**4.Validating the proposed modules into other baseline methods (with more pre-trained models or without pre-training).**

Thanks. We should clarify that our work just focuses on transformer usage in MVS rather than pre-training for MVS. 
MVSFormer[2] has thoroughly discussed the effect of different types of pre-training on MVS. 
To further prove the effectiveness of our proposed method, we provide more detailed experiments in the revision in Table13 of Appendix A.5. 

In Table13, we add our contributions (CVT and SVA) to DINOv1 pre-training based MVSFormer (MVSFormer-P) and CasMVSNet (without pre-training).
We re-train CasMVSNet (CasMVSNet*) as an intermediate baseline for a fair comparison, which contains a 4-stage depth hypothesis (32-16-8-4) and cross-entropy loss, sharing the same setting with MVSFormer and MVSFormer++. 
Since the proposed SVA is a side-tuning module specifically designed for pre-trained models, we only evaluate the effect of SVA on MVSFormer-P. From Table.13, our CVT demonstrates substantial improvements for both CasMVSNet* and MVSFormer-P, and our SVA further enhances the results of MVSFormer-P with CVT. 
These results demonstrate the generalization of all proposed components. We added these discussions to the paper revision.

Reviewer comment (Experiments):
It is advisable to explore more pre-trained models and conduct ablation experiments without pre-trained models. Given DINOv2's frozen state during training, fine-tuning it serves as a pivotal baseline.

Author response:
**3.DINOv2 fine-tuning result as the pivotal baseline.**

Thanks. As discussed in MVSFormer[2], fine-tuning a Plain ViT (DINO, DINOv2) is very costly, especially for high-resolution images. Thus we try to optimize the low-rank fine-tuning (LoRA[3]) rank=16 as a new baseline to verify whether more trainable parameters of the ViT backbone result in prominent improvements (Table10).
The result indicates that more trainable parameters in the backbone only achieve minor improvements. This result also proves the effectiveness of our proposed components, which further push the strong baseline to the limit. 

[2] Cao C, Ren X, Fu Y. MVSFormer: Multi-View Stereo by Learning Robust Image Features and Temperature-based Depth. TMLR2022.
[3] Hu E J, Shen Y, Wallis P, et al. Lora: Low-rank adaptation of large language models[J]. arXiv preprint arXiv:2106.09685, 2021.

Reviewer comment (Presentation):
The paper should include visual comparisons of depth maps to visually demonstrate the accuracy advantages of the estimated depth maps.

Author response:
**5.Missing visual comparisons of depth maps of MVSFormer and MVSFormer++.**

Thanks. We added qualitative comparisons and related discussions for the depth map in Figure9 of Appendix, Our MVSformer++ can estimate more precise depth maps even in challenging scenes.

Reviewer comment (Evaluation):
The paper lacks a discussion of the limitations and failure cases of MVSFormer++. Understanding the method's limitations is crucial for evaluating its real-world applicability.

Author response:
**6.Lacks a discussion of the limitations and failure cases.**

Thanks for this point. We have provided related discussions in Appendix A.8. Though MVSFormer++ enjoys powerful MVS capability as well verified in our experiments, it still suffers from similar limitations as other coarse-to-fine MVS models.
Specifically, the coarse stage struggles for inevitable error estimations for tiny foregrounds in low-resolution inputs, resulting in error accumulations for the following stages as shown in Figure12. Designing a novel dynamic depth interval selection strategy would be a potential solution to handle this problem, which can be seen as interesting future work.

Reviewer comment (Experiments):
What motivated the decision to freeze DINOv2 during pre-training? How does it uniquely contribute to your method? Would including experiments with different pre-trained models or fine-tuning DINOv2 serve as valuable comparisons?

Author response:
**1.Supplemental result of DINOv2 full-model fine-tuning.**

We apologize for the delay in submitting the full-model fine-tuning result of DINOv2, as it requires a significant amount of computation. As mentioned in MVSFormer[2], we carefully set the learning rate of DINOv2 backbone to 3e-5, while all other learning rates for MVS modules are still 1e-3.
All results of DINOv2-based MVS are compared as follows:

| Exp      | Acc    | Comp   | Overall |
|----------|--------|--------|---------|
| DINOv2 (frozen)   | **0.3198** | 0.2549 | 0.2875  |
| DINOv2 (LoRA) | 0.3239 | **0.2501** | **0.2870**  |
| DINOv2 (fine-tuned) | 0.3244 | 0.2566 | 0.2905  |

We would clarify that fine-tuning the whole DINOv2 is costly and achieves inferior performance compared with the frozen one. This phenomenon makes sense. Since DINOv2 is a robust model pre-trained on large-scale curated data, the MVS fine-tuning is based on the limited DTU, which degrades the generalization of DINOv2’s matching capability. 
We added related results and discussion to the Appendix.
Based on these results, our usage of frozen DINOv2 is convincing.

Reviewer comment (Experiments):
Considering the marginal improvement in point cloud metrics, is the increase in network parameters due to adopting DINOv2 justified? How can you demonstrate that the metric enhancements stem from the introduced module's contribution rather than a mere increase in parameters?

Author response:
**2.Cost-benefit analysis of DINOv2. Parameters increasing or module improving?**

Thanks. Since we have accomplished all related experiments of DINOv2, we can prove the improvement does not rely on the cost of more trainable parameters of DINOv2. More importantly, fine-tuning DINOv2 with more trainable parameters could not achieve equivalent improvements as our proposed modules (Overall 0.2905 vs 0.2805). 

On the other hand, our proposed SVA is simplified with fewer self-attention blocks, while normalized 2D-PE, FPE (3D-PE), AAS, adaptive layer scaling and Pre-LN introduce substantial improvements for both depth and point clouds with negligible parameters.
The additional scalability study in Appendix A.2 (Table11) also demonstrates the good efficiency of our model.
So the superior performance of MVSFormer++ relies on reasonable model designs rather than simply using more parameters.

Reviewer comment (Evaluation):
Could you briefly discuss MVSFormer++'s limitations, especially in scenarios where it might underperform?

Author response:
**6.Lacks a discussion of the limitations and failure cases.**

Thanks for this point. We have provided related discussions in Appendix A.8. Though MVSFormer++ enjoys powerful MVS capability as well verified in our experiments, it still suffers from similar limitations as other coarse-to-fine MVS models.
Specifically, the coarse stage struggles for inevitable error estimations for tiny foregrounds in low-resolution inputs, resulting in error accumulations for the following stages as shown in Figure12. Designing a novel dynamic depth interval selection strategy would be a potential solution to handle this problem, which can be seen as interesting future work.

Reviewer comment (Novelty):
Although the MVSFormer++ modifies the base model MVSFormer by DINOv2, SVA, Norm& ALS, FPE, etc, the core contributions share similar designs with other MVS methods.

Author response:
Thanks for this comment. Besides the pre-trained DINOv2 features and highly simplified architecture (only self-attention for reference features), SVA makes a substantial contribution as a "side-tuning module" [1] based on a frozen pre-trained model (refer to Sec 3.1.1, sentence 2). It's crucial to highlight that no gradients are propagated through the frozen DINOv2+SVA, eliminating the need to store the middle tensor of DINOv2 (achieved with torch.no_grad), resulting in significant GPU memory savings. This way is very efficient to incorporate cross-view information into monocular pre-trained vision models. To the best of our knowledge, we are the first to learn cross-view information for DINOv2 with efficient side-tuning. Moreover, we carefully design the normalized 2D-PE for the robustness of various image scales, the position of layer norm (Norm), and adaptive layer scaling (ALS) for stable and better training convergence, pushing the overall performance to the limit (0.2847->0.2805). As the development of large pre-trained models and transformers for 3D vision, our contributions built upon incrementally learning cross-view information for pre-trained ViTs, and detailed transformer designs for feature encoding are beneficial to the MVS community.

[1] Zhang J O, Sax A, Zamir A, et al. Side-tuning: a baseline for network adaptation via additive side networks. ECCV2020.

Reviewer comment (Novelty):
In the feature encoder, the Side View Attention is similar to the Intra-attention and Inter-attention in Transmvsnet. The main differences are that this paper uses a pre-trained DINOv2 as input and removes the self-attention for source features.

Author response:
Thanks for this comment. Besides the pre-trained DINOv2 features and highly simplified architecture (only self-attention for reference features), SVA makes a substantial contribution as a "side-tuning module" [1] based on a frozen pre-trained model (refer to Sec 3.1.1, sentence 2). It's crucial to highlight that no gradients are propagated through the frozen DINOv2+SVA, eliminating the need to store the middle tensor of DINOv2 (achieved with torch.no_grad), resulting in significant GPU memory savings. This way is very efficient to incorporate cross-view information into monocular pre-trained vision models. To the best of our knowledge, we are the first to learn cross-view information for DINOv2 with efficient side-tuning. Moreover, we carefully design the normalized 2D-PE for the robustness of various image scales, the position of layer norm (Norm), and adaptive layer scaling (ALS) for stable and better training convergence, pushing the overall performance to the limit (0.2847->0.2805). As the development of large pre-trained models and transformers for 3D vision, our contributions built upon incrementally learning cross-view information for pre-trained ViTs, and detailed transformer designs for feature encoding are beneficial to the MVS community.

[1] Zhang J O, Sax A, Zamir A, et al. Side-tuning: a baseline for network adaptation via additive side networks. ECCV2020.

Reviewer comment (Novelty):
The use of linear attention in the feature encoder has already been proposed in Transmvsnet.

Author response:
Thanks for this point. Although Transmvsnet leveraged linear attention for feature learning, they involved a compromise to save the attention computation, which leaks in-depth discussion about transformer usage in MVS. In our study, we thoroughly investigate the impact of various attention mechanisms for different MVS parts, as outlined in Table 5. As shown in Table5(right), linear attention became the best choice of **feature encoding** because of the global respective fields and image scaling robustness as discussed in Sec.3.1.1 and Sec4.2.3. We have also revealed an insightful finding that **linear attention enjoys good performance in feature encoding, but it performs very terribly in cost volume (Table5 left)**. Since features in cost volume are built upon feature similarity, leaking the essentially informative feature presentation, feature aggregating-based linear attention cannot handle the cost volume regularization compared with spatial aggregating-based attention (discussed in Sec3.2.1 paragraph2). We hope that such an insightful contribution could benefit the community for a better combination of transformer and MVS learning.

Reviewer comment (Evaluation):
In Table 9, although with a larger network, the MVSFormer++ only improves on MVSFormer by a small margin, which can not fully support the claim of the effectiveness of 2D-PE and AAS.

Author response:
Thanks. We need to clarify that our ETH3D results are all achieved with the same threshold (0.5) of depth confidence filter and default settings of dynamic point cloud fusion (DPCD)[2] without any cherry-pick hyper-parameter adjusting. This approach ensures a fair and unbiased evaluation. Though improvements may not be very significant, they are consistent and indicative of the robustness and competitiveness of MVSFormer++.  We have claimed these points in the paper revision. 

[2] Yan J, Wei Z, Yi H, et al. Dense hybrid recurrent multi-view stereo net with dynamic consistency checking. ECCV2020.

Reviewer comment (Reproducibility):
The FPE in Table 4 shows good improvement on CVT. The detailed network structure should be made more clear. Please see the questions.

Author response:
Thanks for this valuable comment. We have to clarify both CVT and FPE are only used in the first stage (1/8 resolution) as introduced in Sec3.2.1 paragraph2. So CVT+FPE is only applied to the first stage in Table4. As discussed in Sec.3.2.1, we find that CVT for other stages would cause inferior performance. Because only the first stage in a cascade model enjoys a complete and continuous 3D scene. Therefore, the integrality and continuity of the cost volume are very important for CVT. Although FPE is the key to unlocking the capacity of CVT (like position encoding for Transformer), for the 3DCNN-based cost volume regularization of other stages(>1), FPE is not necessary. Because CNN with zero padding could provide sufficient position clues[3]. We further clarify this in the paper revision.

[3] Islam M A, Jia S, Bruce N D B. How much position information do convolutional neural networks encode? ICLR2020.

Reviewer comment (Experiments):
The evidence for the minor changes such as the LN and AAS is not strong with experiments on only DTU. They are more intuitive and may need more experiments to prove whether they are generalizable designs. For example, Table 9 on ETH3D actually cannot fully support AAS.

Author response:
Thanks for this valuable feedback. We appreciate the reviewer's suggestion for additional experiments to further establish the effect of the proposed components, including Normalized 2D-PE, FPE, and AAS. To further prove the effect of our proposed methods for handling high-resolution scenes, we detail additional ablation studies to Table12 in Appendix A.3 with different image sizes and discuss their influence on MVS depth estimation. 
Both high-resolution depth and point cloud results outperform the low-resolution ones, which proves the importance of adaptability for high-resolution images. Moreover, FPE and AAS improve the results under both 576\*768 and 1152\*1536 images. However, AAS is more effective for depth estimation in high-resolution cases, while the depth gap in low-resolution ones is not obvious. The normalized 2D-PE plays a very important role in high-resolution feature encoding, contributing substantial improvements in both depth and point clouds.
We have also added these results and discussions to the paper revision.
For the LayerNorm and adaptive layer scaling, we clarify that these techniques are important to stabilize the SVA training based on frozen DINOv2, which has already been verified clearly in Table4 and Table6.

Reviewer comment (Experiments):
I would like to know the detailed structure differences between CVT and CVT+FPE. CVT is only used in the first coarse stage so how many stages use the CVT+FPE in Table 4? What are the results when CVT and CVT+FPE are both used in all stages or only the first coarse stage?

Author response:
Thanks for this valuable comment. We have to clarify both CVT and FPE are only used in the first stage (1/8 resolution) as introduced in Sec3.2.1 paragraph2. So CVT+FPE is only applied to the first stage in Table4. As discussed in Sec.3.2.1, we find that CVT for other stages would cause inferior performance. Because only the first stage in a cascade model enjoys a complete and continuous 3D scene. Therefore, the integrality and continuity of the cost volume are very important for CVT. Although FPE is the key to unlocking the capacity of CVT (like position encoding for Transformer), for the 3DCNN-based cost volume regularization of other stages(>1), FPE is not necessary. Because CNN with zero padding could provide sufficient position clues[3]. We further clarify this in the paper revision.

[3] Islam M A, Jia S, Bruce N D B. How much position information do convolutional neural networks encode? ICLR2020.

Reviewer comment (Evaluation):
In the ablation study, the results of Norm&ALS under the depth error ratios of 2mm and 4mm are slightly inferior.

Author response:
**1.Depth results of 2mm and 4mm are inferior with Norm&ALS.**

Thanks for this point. We have to clarify that depth annotations of DTU are not complete and absolutely correct as discussed in [1]. Thus the results of the point cloud should be more convincing. Moreover, Norm&ALS is mainly working for more stable feature learning in low-resolution (DINOv2 feature), so it results in fewer large-depth errors. This result conforms to the expectation of DINOv2+SVA which learns more robust low-resolution depth, while precise high-resolution depth should be learned by convolutions. Better results of point clouds also prove our aforementioned opinion.

[1] Luo K, Guan T, Ju L, et al. Attention-aware multi-view stereo CVPR2020.

Reviewer comment (Evaluation):
A discussion regarding the limitations is missing.

Author response:
**2.Missing limitation discussion.**

Thanks for this point. We have provided related discussions in Appendix A.8. Though MVSFormer++ enjoys powerful MVS capability as well verified in our experiments, it still suffers from similar limitations as other coarse-to-fine MVS models.
Specifically, the coarse stage struggles for inevitable error estimations for tiny foregrounds, resulting in error accumulations for the following stages as shown in Figure12. Designing a novel dynamic depth interval selection strategy would be a potential solution to handle this problem, which can be seen as interesting future work.

Reviewer comment (Presentation):
Minor: Section 4.1 Experimental performance, mean F-score is 41.75 on the Advanced sets in the text while in Tab.3 it is 41.70.

Author response:
**3.Wrong text description.**

We apologize for the typo here. We revised the result to 41.70 in the revision.

Reviewer comment (Experiments):
Except for the customized designs beyond the MVSFormer, this work leverages DINOv2 as a new backbone (compared to DINO used in the MVSFormer). It would be interesting to see how the performance of MVSFormer++ changes when it keeps the same backbone as that of MVSFormer.

Author response:
**1.Performance changing of DINOv2 backbone in MVSFormer++; does the improvement gain from a strong backbone?**

Thanks for this comment. The baseline result of MVSFormer enhanced by frozen DINOv2 is listed in the first row of Table4. We further compare the baseline of MVSFormer+DINOv2 with MVSFormer as follows (Table10 of the revision):

| ViT backbone | Frozen backbone | Accuracy$\downarrow$ | Completeness$\downarrow$ | Overall$\downarrow$ |
|--------------|-----------------|----------------------|--------------------------|------------------------------------|
| DINO-small   | $\checkmark$    | 0.327                | 0.265                    | 0.296               |
| DINO-base    | $\checkmark$    | 0.334                | 0.268                    | 0.301               |
| Twins-small  | $\times$        | 0.327                | 0.251                    | 0.289               |
| Twins-base   | $\times$        | 0.326                | 0.252                    | 0.289               |
| DINOv2-base  | $\checkmark$    | **0.3198**     | 0.2549                   | 0.2875              |
| DINOv2-base  | LoRA (rank=16)  | 0.3239               | **0.2501**          | **0.2870**     |

Compared with DINO and Twins, MVSFormer based on DINOv2 enjoys slightly better results with a frozen ViT backbone. This is strengthened by robust visual features from DINOv2 verified as zero-shot cross-domain feature matching experiments in the DINOv2 paper[1]. We further clarify the motivation for selecting DINOv2 in the last of Appendix A.1. Although MVSFormer+DINOv2 is a strong baseline, we need to clarify that our work is orthogonal to the DINOv2. DINOv2 just improved the overall distance from 0.289 to 0.2875 with less computation (frozen backbone compared with Twins), while we further improved the overall distance from 0.2875 to 0.2805 with our novel designs of SVA, CVT, and image scaling adaption. To verify the effectiveness of our contribution, we further show the improvements of the proposed components based on MVSFormer-P (DINOv1) and CasMVSNet as in Tab.13 of Appendix A.5.
Our CVT demonstrates substantial improvements for both CasMVSNet* and MVSFormer-P, and our SVA further enhances the results of MVSFormer-P with CVT.
We added these discussions to the paper revision.

[1] Oquab M, Darcet T, Moutakanni T, et al. Dinov2: Learning robust visual features without supervision[J]. arXiv preprint arXiv:2304.07193, 2023.

Reviewer comment (Evaluation):
MVSFormer and MVSFormer++ show different reconstruction performances regarding different cases on Tanks-and-Temples (Table 3). The authors are suggested to provide more discussions on how the qualitative results differ (like local details and global distributions) and why the degenerations happen.

Author response:
**3.Discuss the qualitative results on Tanks-and-Temples.**

Thanks. We have compared more qualitative results between MVSFormer and MVSFormer++  in Figure7 of the Appendix. In general, MVSFormer++ achieves much more precise geometric reconstruction (better precision), while MVSFormer shows more complete results in some scenes, such as the “Recall” subfigure of “Playground”. For the trading-off between precision and recall, MVSFormer++ obviously enjoys a superior balance.
We added these discussions to the paper revision Appendix A.7.

Reviewer comment (Evaluation):
The performance of the complete version of this work in the ablation study is different from the quantitative results reported in Table 2. Please elaborate on this inconsistency in metrics.

Author response:
**4.Different results of the full model in Table2 and ablation study.**

We apologize for the confusing results. Both results of MVSFormer++ in the last row of Table2 and ablations share the same model setting, i.e., our full model. The performance difference is just randomness from two different training results. Because of the server allocation issue, we re-trained the model for the ablation, and subsequently fine-tuned it for additional experiments such as Tanks-and-Temples and ETH3D. The outcomes of them are detailed below. The differences are minimal and can be attributed to the inherent randomness in the training process. 

| Exp      | Acc    | Comp   | Overall |
|----------|--------|--------|---------|
| Table2(old)   | 0.3105 | 0.2503 | 0.2804  |
| Ablation(new) | 0.3090 | 0.2521 | 0.2805  |

We have revised the results in Table2, and appreciate your careful reading.

Reviewer comment (Experiments):
The baseline version of MVSFormer (without CVT, FPE, AAS, SVA, Norm&ALS) seems kind of strong already. Does it gain from the strong backbone? Moreover, the qualitative results of the ablation study are expected to be provided.

Author response:
**1.Performance changing of DINOv2 backbone in MVSFormer++; does the improvement gain from a strong backbone?**

Thanks for this comment. The baseline result of MVSFormer enhanced by frozen DINOv2 is listed in the first row of Table4. We further compare the baseline of MVSFormer+DINOv2 with MVSFormer as follows (Table10 of the revision):

| ViT backbone | Frozen backbone | Accuracy$\downarrow$ | Completeness$\downarrow$ | Overall$\downarrow$ |
|--------------|-----------------|----------------------|--------------------------|------------------------------------|
| DINO-small   | $\checkmark$    | 0.327                | 0.265                    | 0.296               |
| DINO-base    | $\checkmark$    | 0.334                | 0.268                    | 0.301               |
| Twins-small  | $\times$        | 0.327                | 0.251                    | 0.289               |
| Twins-base   | $\times$        | 0.326                | 0.252                    | 0.289               |
| DINOv2-base  | $\checkmark$    | **0.3198**     | 0.2549                   | 0.2875              |
| DINOv2-base  | LoRA (rank=16)  | 0.3239               | **0.2501**          | **0.2870**     |

Compared with DINO and Twins, MVSFormer based on DINOv2 enjoys slightly better results with a frozen ViT backbone. This is strengthened by robust visual features from DINOv2 verified as zero-shot cross-domain feature matching experiments in the DINOv2 paper[1]. We further clarify the motivation for selecting DINOv2 in the last of Appendix A.1. Although MVSFormer+DINOv2 is a strong baseline, we need to clarify that our work is orthogonal to the DINOv2. DINOv2 just improved the overall distance from 0.289 to 0.2875 with less computation (frozen backbone compared with Twins), while we further improved the overall distance from 0.2875 to 0.2805 with our novel designs of SVA, CVT, and image scaling adaption. To verify the effectiveness of our contribution, we further show the improvements of the proposed components based on MVSFormer-P (DINOv1) and CasMVSNet as in Tab.13 of Appendix A.5.
Our CVT demonstrates substantial improvements for both CasMVSNet* and MVSFormer-P, and our SVA further enhances the results of MVSFormer-P with CVT.
We added these discussions to the paper revision.

[1] Oquab M, Darcet T, Moutakanni T, et al. Dinov2: Learning robust visual features without supervision[J]. arXiv preprint arXiv:2304.07193, 2023.

**2.Qualitative results of the ablation study**

Thanks for this point. We have compared the qualitative ablation study in Appendix Figure8.  From Figure8, CVT and AAS could effectively eliminate the outliers, while SVA with normalized 2D-PE is also critical for precise point clouds.

Reviewer comment (Writing):
The description of Normalization and Adaptive Layer Scaling is ambiguous and unclear. More details about the motivation and implementation would be helpful to understand this part.

Author response:
**5.More details about motivation and implementation of Normalization and Adaptive Layer Scaling.**

Thanks for this comment. For the normalization, we have discussed Pre-LN and Post-LN in the related work of "LN and PE in Transformers''. We provide more details about them in Appendix A.1. Post-LN is the regular setting of transformer, which normalizes features after the residual addition, while Pre-LN normalizes them before the attention and Feed Forward Network (FFN).

Post-LN: x=LN(x+attn(x)), x=LN(x+ffn(x)).

Pre-LN: x=x+attn(LN(x)), x=x+ffn(LN(x)).

For the MVS encoding, we found that Pre-LN enjoys more significant gradient updates, especially when being trained for multi-layer attention blocks (as discussed in [2]). In Table6, Pre-LN enjoys superior performance, while we find that Post-LN usually results in slower convergence.
For Adaptive Layer Scaling (ALS), we have analyzed the high variance issue in Figure6(a) of the Appendix, and multiplied learnable coefficients to different layers' features of frozen DINOv2. Thus our model could adaptively learn the significance of different DINOv2 layers, which stabilize the MVS training. 
We have added all related presentations in the paper revision to make our idea clearer for readers.

[2] Wang Q, Li B, Xiao T, et al. Learning deep transformer models for machine translation. ACL2019.