Paper: LEAP: Liberate Sparse-View 3D Modeling from Camera Poses (ICLR-2024)

Reviewer comment (Experiments):
The authors claim that they significantly outperform prior methods on scene-scale datasets. However, LEAP only presents a comparison on the small DTU dataset. This comparison is somewhat flawed, since it prevents the authors from benchmarking against SRT, which is focused on scene-scale novel view synthesis. It would be much more convincing to see results on MultiShapeNet, which is a publicly available scene-scale dataset that SRT is trained and compared on. Showing more scene-scale results seems particularly important because LEAP's use of a 3D voxel grid with fixed extents seems likely to limit its usefulness on scene-scale datasets. It is also worth noting that the scenes in the DTU dataset have small camera baselines, meaning that classical pose estimation would produce near-perfect estimated poses on DTU. Using a wide-baseline dataset for scene-scale comparison (where pose estimation is more likely to fail) would be more convincing.

Author response:
**[Comparison with SRT]** Thanks for the suggestion. We described the limitations of LEAP on unbounded scenes on page 9 of our paper (asked by the reviewer). We further clarify the differences with SRT: i) **LEAP targets at 3D modeling** -- it uses a *3D representation* that can export the density field and render depth (with evaluation results shown below). Dealing with unbounded scenes is a long-standing problem with 3D-aware methods [1], which is not our current focus. **SRT targets at novel view synthesis** based on *2D representations*. It can be applied to unbounded scenes, but it cannot produce 3D geometry directly. To address your concern, we will specify the scene-level results to bounded scenes as you suggested; ii) Moreover, the absence of SRT in DTU experiments is because *SRT requires large training data* and it cannot adapt to the small DTU data – It is a property of SRT rather than intentionally excluding it. In contrast, LEAP works better with limited training data. All comparisons of objects and bounded scenes are fair and extensive, which is acknowledged by all other reviewers and demonstrates the effectiveness of LEAP. We will better clarify these points in the revision.

|               | FORGE | LEAP |
|:-----------:|:------------:|:------------:|
| Depth err. | 0.16      | **0.11**         | 


[1] Barron, Jonathan T. et al. “Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields.” CVPR 2022.

---

**[DTU details]** Pose estimation on DTU is not solved. As shown in Fig 7 of SPARF, the pose errors are still large with sparse inputs.

Reviewer comment (Evaluation):
Figure 8 appears to be flawed and somewhat misleading. The experiment setup, as I understand it, is that the authors feed a synthetic scene consisting of a single dot projected onto five images into a trained LEAP network. The authors show that the neural volume's density is highest along the ray that corresponds to the dot's location in the canonical view. This is good evidence that the network has "understood" the meaning of the canonical view and has learned to map pixels to rays within the volume. However, the authors claim that the network then "leverages the multi-view information to resolve the depth ambiguity of the ray." However, estimating pose given images of a single dot is impossible, since the scene's appearance is invariant to rotation around the dot. Since pose and correspondence are both needed to correctly place a 3D point, this means that in this case, the network cannot actually be doing what the authors claim. It seems much more likely that because the training dataset consisted of objects/surfaces that were all at roughly the same depth, the network is simply placing the dot near the mean of those depths.

Author response:
**[Fig 8]** The location of the point visualized in the volume clearly matches the GT views. Simply placing the density near the mean value will not match the GT. Besides, the reviewer mentions “estimating pose given images of a single dot is impossible”. However, LEAP doesn’t estimate any poses. To further address your concern, another piece of evidence is in Fig 16, where LEAP shows large depth errors with one input, and it is almost resolved with two inputs, verifying LEAP uses multiview information to resolve depth ambiguity.

Reviewer comment (Novelty):
The authors cannot both claim that their method is the first to introduce the pose-free paradigm (first paragraph, page 4) while also stating that the SRT already introduced the same paradigm (which it very much did). The authors should soften that claim to saying that they merely implement a new way of solving this pose-free problem, rather than introducing it.

Author response:
**[The pose-free approach]** We never claim we are “the first” in the first paragraph of page 4 (asked by the reviewer). What we claim is “a novel pose-free paradigm” – LEAP is indeed different from SRT as it is based on 3D representations for 3D modeling. We will make this more clear. We give credit to SRT, which is the first pose-free work and it performs novel view synthesis based on 2D representations.

Reviewer comment (Theory):
The authors state that the 2D-to-3D step happens in a coarse-to-fine manner, but do not describe how. It seems that the network emergently shows coarse-to-fine behavior (boundaries getting sharper during successive lifting attention layers), not that there's an upscaling operation between lifting attention layers.

Author response:
**[Coarse-to-fine 2D-to-3D mapping]** Your understanding is correct. It is a learned behavior. We will clarify it.

Reviewer comment (Writing):
The paper contains a large number of typos and grammatical errors that need to be fixed (too many to list).

Author response:
**[Typos and figure color]** We will revise them. Thanks for the detailed review!

Reviewer comment (Presentation):
In figure 2, the colors for fusion with high weight and fusion with low weight are extremely similar when printed.

Author response:
**[Typos and figure color]** We will revise them. Thanks for the detailed review!

Reviewer comment (Novelty):
It would be worth mentioning "RUST: Really Unposed SRT," "View Matching Neural Radiance Fields," and "GAN-Based Neural Radiance Field Without Posed Camera" in the related works section.

Author response:
**[Related work]** We will cite and add more discussion on the mentioned works. RUST resolves the necessity of target camera pose by prompting the model with a partial target image. VMRF and GNeRF focus on the dense-view setting.

Reviewer comment (Writing):
Clarity in the exposition of the method's approach to generating predicted images during the optimization process would be beneficial.

Author response:
**[Render input views during training/optimization]** Different from the original NeRF-style works, where the model is trained and tested on the same instance, LEAP deals with modeling **novel instances captured as sparse & unposed images at inference**. It is *trained and tested on different object/scene instances*, and all test instances are unseen during training. As described in the third paragraph of page 2, we use **ground-truth poses** of input images of training scenes to render the predicted input views. Note that these ground-truth poses are **only used during training** for learning the 2D-3D information mapping, and using ground-truth pose during training is a **common setting** in prior generalizable NeRF works, e.g. PixelNeRF, SRT, and FORGE. Besides, we note that the ground-truth poses used during training are *easily accessible*, as they are provided in the existing datasets, e.g. OmniObject3D, Objaverse, and DTU dataset, annotated using virtual cameras for rendering or complicated camera calibration techniques. After LEAP is trained, it directly **inferences on** novel objects/scenes captured as **unposed images** with **zero-shot generalization**. We will better clarify the point in the revision.

Reviewer comment (Evaluation):
It would be advantageous if the paper delved deeper into the reasons behind its enhanced speed and the trade-offs involved in achieving such acceleration.

Author response:
**[Improved speed]** The fast inference speed of LEAP originates from its ability to predict the radiance field in **a single feed-forward step** during inference, without any test-time optimization (as we described in the first paragraph of page 2). In contrast, most prior works require some per-scene optimization at test time (SPARF, RelPose, and FORGE), which are slow to converge. We will clarify it in the revision.

Reviewer comment (Reproducibility):
Concerning the proposed methodology, the process of generating predictions remains unclear. Specifically, after enhancing the 3D neural volume, how is the establishment of a 2D to 3D association executed for rendering the input view during optimization? Regrettably, this paper does not provide a satisfactory response to this inquiry.

Author response:
**[Render input views during training/optimization]** Different from the original NeRF-style works, where the model is trained and tested on the same instance, LEAP deals with modeling **novel instances captured as sparse & unposed images at inference**. It is *trained and tested on different object/scene instances*, and all test instances are unseen during training. As described in the third paragraph of page 2, we use **ground-truth poses** of input images of training scenes to render the predicted input views. Note that these ground-truth poses are **only used during training** for learning the 2D-3D information mapping, and using ground-truth pose during training is a **common setting** in prior generalizable NeRF works, e.g. PixelNeRF, SRT, and FORGE. Besides, we note that the ground-truth poses used during training are *easily accessible*, as they are provided in the existing datasets, e.g. OmniObject3D, Objaverse, and DTU dataset, annotated using virtual cameras for rendering or complicated camera calibration techniques. After LEAP is trained, it directly **inferences on** novel objects/scenes captured as **unposed images** with **zero-shot generalization**. We will better clarify the point in the revision.

Reviewer comment (Writing):
A noticeable absence of local consistency, which is essential for ensuring a robust 3D to 2D association, prompts the question of how the proposed approach manages challenges such as occlusion and variations in lighting conditions. Clarity regarding the method's strategy for addressing these issues would be greatly appreciated.

Author response:
**[Local consistency]** LEAP learns the local consistency between 2D-3D. As shown in Fig. 11 (a) and (b), the *two neighbor on-surface 3D voxels are associated with consistent 2D regions*. We generally agree with your point that incorporating more regularizations can improve the 2D-3D association accuracy.

Reviewer comment (Experiments):
The impact of an increasing number of input views on the method's prediction accuracy compared to baseline methods remains unaddressed. It is crucial to understand whether the accuracy is expected to decrease or improve with a greater quantity of input views. The author's insights on this matter would be valuable.

Author response:
**[Increasing number of views]** As shown in Fig. 8 (bottom), we provide the performance of LEAP compared with prior works using 2-5 input views, where all methods are trained with 5 images of each scene/object. LEAP demonstrates better performance than prior works when more images are available. Besides, we note that LEAP focuses on *sparse-view reconstruction*, where we usually have only 2-5 input views. To further address your concern, we experiment with using 7 and 10 input views, which reaches the upper bound of the number of views in prior sparse-view research [2]. As shown below, the performance of the best baseline FORGE drops with more than 7 views. We conjecture that the reason is the compounding pose estimation error. In contrast, LEAP continually demonstrates better performance with more inputs.

|               | 5 Views | 7 Views | 10 Views |
|:-----------:|:------------:|:------------:|:------------:|
| FORGE | 26.56       | 27.16         | 27.08         |
| LEAP     | **29.10**      | **29.75**         | **30.11**         |

[2] Zhang, Jason Y. et al. “RelPose: Predicting Probabilistic Relative Rotation for Single Objects in the Wild.” ECCV 2022.

Reviewer comment (Evaluation):
Unlike the traditional pose-based projection, the proposed 2D-3D mapping layers are a weighted fusion of 2D features. The mapping may be more robust than pose-based projection when the pose is inaccurate but it limits in accuracy. Consequently, the reconstruction and rendering are often blurred, as shown in Figure 6. Do authors have ideas to improve it?

Author response:
**[Performance upper bound]** Your understanding is correct. Using the pose-free approach is a better strategy for handling pose estimation errors. However, we argue that the pose-free approach does not limit its accuracy, as LEAP is better than prior generalizable NeRFs, i.e., PixelNeRF and FORGE when they use estimated poses. It’s reasonable that the performance is upper-bounded by the results with GT poses, and actually LEAP is better than PixelNeRF+GT poses in some cases. The mentioned blurry results in Fig. 6 are caused by another problem, and we discuss it as follows.

---

**[Reconstruction quality]** We appreciate your detailed observation. We note that the result in Fig. 6 (DTU experiments) is a test of LEAP on handling **small-scale training data**. In detail, the DTU dataset only contains 88 scenes for training, and all test scenes are unseen during training. Thus, the performance in Fig. 6 is not so “perfect”, which is caused by the limited training data scale rather than the pose-free framework itself. When using training datasets with normal scales (e.g. > 1K training instances), LEAP demonstrates good enough performance, e.g., shown in Fig. 4, Fig. 5, Fig. 12, and Fig. 13 for Omniobject3D, ShapeNet, and Objaverse, respectively. Improving the performance with limited training data is an open problem for deep learning-based methods. We believe that applying better transfer learning strategies on models pre-trained with normal-scale or larger-scale data can improve the performance in this setting. Besides, we believe applying more geometry-level regularizations on the refined neural volume and predicted radiance field can improve the performance.

Reviewer comment (Experiments):
Sparse-view reconstruction is an ill-posed problem because the input images contain incomplete scene information. Although the proposed method is better than previous approaches, the performance is still very limited. In this scenario, most of the existing methods can refine the results by using more input images. However, the proposed method only aggregates information and runs forward pass at the inference time. Is it possible to do refinement when more input images are given?

Author response:
**[Refine results with more incoming images]** We appreciate the valuable suggestion. The mentioned setting of refining the prediction with more incoming images is a very practical problem, as is the incremental reconstruction [1]. To address your concern, we provide a study using 5 existing views with 1 incoming view. Our solution is i) reconstructing the neural volume using the 5 existing views as an initialization, ii) using the multiview encoder to propagate information between the canonical view and the incoming view, and iii) refining the existing neural volume using the image features of the incoming and canonical view by performing 2D-3D information mapping attentions. Using the refined neural volume for predicting the radiance field demonstrates 0.37 PSNR novel view synthesis improvement, compared with the results from the existing 5 views.  For reference, direct inference on 6 images observes a 0.45 PSNR improvement. The slight gap shows the feasibility of the solution and the potential of LEAP in handling the incremental reconstruction problem. Besides, we believe the mentioned task is a promising research direction where a lot of details should be considered (while it is beyond the current scope of this paper). For example, whether the input views are temporally correlated, i.e., video frames, or whether the lighting conditions have been changed in incoming views. We thank you for the insightful suggestions and we plan to explore it further in the future.

[1] Yuan, Ze-Huan and Tong Lu. “Incremental 3D reconstruction using Bayesian learning.” Applied Intelligence 2012.

Reviewer comment (Experiments):
Does the proposed method work well with different frame numbers during inference time? As the proposed 2d-3d mapping needs aggregating information from all frames, does it consume huge memory in dense views (e.g., more than 100 frames)?

Author response:
**[Inference on different numbers of inputs]** Our method is able to work on different numbers of inputs. As shown in Fig. 8 (bottom), LEAP is able to work with fewer images (2-4 views) when the model is trained with 5 views. We further evaluate the results using 7 and 10 images (as shown below), which reach the upper bound of the number of views in prior sparse-view research [1]. Under the dense view setting (>100 images as mentioned), the pose estimation accuracy will not be a problem, and, thus, we can perform pose-aware reconstruction, e.g. COLMAP. For extending LEAP to dense views, as you mentioned, it is necessary to propose hierarchical information aggregation techniques, as the aggregation from all input views can be computationally expensive. We note that working on dense views is beyond the current scope of LEAP and it's an interesting open research problem to unify sparse-view and dense-view reconstruction methods. We appreciate your insightful comments!

|               | 5 Views | 7 Views | 10 Views |
|:-----------:|:------------:|:------------:|:------------:|
| FORGE | 26.56       | 27.16         | 27.08         |
| LEAP     | **29.10**      | **29.75**         | **30.11**         |

[1] Zhang, Jason Y. et al. “RelPose: Predicting Probabilistic Relative Rotation for Single Objects in the Wild.” ECCV 2022.

Reviewer comment (Reproducibility):
Novel view synthesis is defined as rendering the images as specific camera pose and time (for dynamic scenes). When the camera poses are not avaliable or not estimated as this paper, how to deal with NVS with given camera poses, i.e., how to align the given camera poses with the training set images.

Author response:
We use “relative” camera poses for both training and inference. The relative camera pose specifies the rotation and translation of a target view with respect to the canonical view (first image) [1, 2]. Thus, during testing, you can control the target NVS viewpoints by specifying small or large relative camera transformations. This definition of target NVS camera pose will not be influenced by the different definitions or absence of “absolute” camera pose of input views, as the reconstructed radiance field is defined in the local frame of canonical view. Furthermore, we follow prior works [3] to perform reconstruction with normalized scales, which eliminates the scale ambiguity of camera translations caused by RGB-only observations. We will include more details in the revision and we thank you for your helpful feedback. Additionally, we show 360-degree visualization in the supplementary by controlling the relative poses of target NVS views.

[1] Zhang, Jason Y. et al. “RelPose: Predicting Probabilistic Relative Rotation for Single Objects in the Wild.” ECCV 2022.
[2] Jiang, Hanwen et al. “Few-View Object Reconstruction with Unknown Categories and Camera Poses.” 3DV 2024.
[3] Schönberger, Johannes L. and Jan-Michael Frahm. “Structure-from-Motion Revisited.” CVPR 2016.

Reviewer comment (Experiments):
Essentially, the method incoorporated the feature correspondences into the overall optimization, it is thus interesting to make comparisons with estimated correspondences from optical flow where the dense matching is learned.

Author response:
We generally agree that more explicit use of fine-grained cross-view correspondences/matching can facilitate learning. However, the existing correspondence models are not suitable for finding correspondence in the sparse-view setting. For example, the mentioned *optical flow methods* typically work on nearby frames in a video, which observes *slight visual appearance differences*. In contrast, input images in the *sparse-view setting* usually have *extremely large variations*, caused by *wide-baseline cameras*. The current optical flow methods cannot work well in this setting [4]. Besides, one of our baseline methods, SPARF, is based on dense correspondence from off-the-shelf methods, which shows strong artifacts when correspondence is not accurate and suffers from slow convergence. Finally, we believe pixel-level dense correspondence and sparse-view reconstruction with feature-level correspondence can benefit from each other. Exploring their synergy to improve the performance of both, i.e., using a joint learning framework, can be a promising future direction. We appreciate your insightful comments.

[4] Chen, Qiao and Charalambos (Charis) Poullis. “Motion estimation for large displacements and deformations.” Scientific Reports 12 2022.

Reviewer comment (Experiments):
It is worth to compare related work such as Unposed NERF to verify the effectiveness of the proposed method.

Author response:
We clarify that this paper focuses on *sparse-view reconstruction*, where only 2-to-5 unordered images are available. In contrast, the current unposed NeRFs usually assume either dense-view (~50) or sequential (video) inputs [5,6]. This line of work demonstrates limited performance in the sparse-view setting, as they still estimate camera poses internally. We run experiments on the DTU dataset with Nope-NeRF [5] under the sparse-view setting, where its PSNR is 11.32. This performance gap (compared with their satisfying performance using dense-view inputs) verifies the necessity of this work. Besides, one of our baselines, FORGE, is an unposed & generalizable NeRF variant designed for sparse views, and LEAP demonstrates better performance compared with FORGE. We will add more discussion on this in the revision. 

[5] Bian, Wenjing et al. “NoPe-NeRF: Optimising Neural Radiance Field with No Pose Prior.” CVPR 2023.
[6] Fu, Yang et al. “MonoNeRF: Learning Generalizable NeRFs from Monocular Videos without Camera Poses.” ICML 2023.