Paper: PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Prediction (ICLR-2024)

Reviewer comment (Experiments):
I have a suspicion that the baselines - RelPose++ and Forge - are not trained on the same datasets, but that the authors are rather using the pre-trained methods and just evaluate them on these datasets. Could the authors clarify?

Author response:
**Evaluating RelPose++ and Forge**: we used the pretrained checkpoints from RelPose++ and Forge to evaluate their cross-dataset generalization. We would like to emphasize that we focus on cross-dataset generalization in this work as it is vital for practical applications; our model was not trained on any of the evaluation datasets either. Moreover, we would like to highlight that we also reconstruct NeRF while RelPose++ doesn’t. This joint prediction of 3D and pose is very helpful for boosting pose prediction accuracy in our experiments, as shown in Tab. 4.

Reviewer comment (Experiments):
One baseline is very clearly missing - the Scene Representation Transformer, Sajjadi et al, which can similarly perform novel view synthesis from unposed images at test time while requiring camera poses at training time.

Author response:
**Missing Scene representation transformer (SRT) baseline**:  In our humble opinion, we can not consider SRT as a fair baseline to be compared, since there’s a clear distinction in objectives and scopes between our work and SRT. PF-LRM targets 3D reconstruction and pose estimation, while SRT only focuses on novel view synthesis. It means that SRT does not estimate poses for input images which is one of the main goals of this paper and densely measured. Poses are important for practical applications like robotic navigation, etc. Also, geometry can be naturally extracted from our reconstructed NeRF, while SRT is a geometry-free method that does not directly predict geometry. However, we thank the reviewer for bringing this work to our attention, we will cite SRT and discuss it in a revision.

Reviewer comment (Experiments):
The prior point is even more critical since the authors don't always outperform RelPose++ on Co3D. If RelPose++ wasn't trained at a similar scale, this is concerning.

Author response:
**Comparison with RelPose++ on Co3D**: as shown in Tab. 8 of the appendix, we outperform RelPose++ on the majority of the Co3D categories, even though our model is not trained on Co3D while RelPose++ was trained on it. As to why we are worse on certain categories, we attribute this to the noisy in-accurate masks provided by Co3D, which we used to remove background before running our method. A clear performance drop can also be seen in Tab. 8 when we test RelPose++ on such background-free images. We think handling background in our framework is an interesting future direction.

Reviewer comment (Evaluation):
In the evaluation section, the authors state that they evaluate PSNR against the input images. That is questionable - they should evaluate the PSNR against held-out test views?

Author response:
**Input-view PSNR**: We will include novel-view psnr in a revised draft before the rebuttal deadline.  The current input-view psnr is to show that our predictions (both NeRF and poses) are well-aligned with the input images, as we render our predicted NeRF using our predicted poses (in Fig. 3); we would like to highlight that we do well in this important objective of matching re-renderings with inputs for joint 3D and pose prediction, while baselines struggle with this goal.  We also show extracted geometry in Fig. 3, 4 and interactive NeRF viewers on our project page to demonstrate that we did not cheat by predicting degenerate solutions. In Fig. 4, we show re-rendering of our reconstructed NeRF using ground-truth pose, to demonstrate that removing pose prediction from our model leads to lower reconstruction quality. This is to highlight the synergy between NeRF reconstruction and pose estimation, which was not shown in prior work like SRT.

Reviewer comment (Writing):
The authors should clarify - already in the introduction and the abstract - that their method still requires ground-truth poses at training time. I.e., it is only pose-free at test time!

Author response:
Updated draft: we have updated (in-place) the abstract, introduction, related work, conclusion sections. We’ve mentioned in the updated abstract that we used multi-view posed data for training. We’ve cited SRT paper and discussed the scope difference between our work and SRT. We also added citations to the Convolutional Occupancy Networks paper and FlowCam paper, as the reviewer suggested. For the experiment section, we did not modify it in-place because our modifications will change the figure/table numbers, hence increasing the reading difficulty for the reviewers. Instead, we put the updated experiment section in Appendix B. We plan to integrate it upon acceptance. We are also happy to integrate it now, if the reviewers don’t think this increases reading burden. You can find our novel-view PSNR in Tab. 14 and our novel-view renderings (along with ground-truth) in Fig. 9. We also include a detailed limitation section in Appendix C.

Reviewer comment (Experiments):
The authors should demonstrate their method on some real-world captures instead of only synthetic examples.

Author response:
**Missing real-data results in the paper**: We respectfully disagree with this claim. We have shown our model’s results on the real DTU dataset (Fig 5 and Tab 9 in appendix), OmniObject3D dataset (Tab 1), Co3D dataset (Tab 8 in appendix). Please refer to our main paper and appendix for details.

Reviewer comment (Novelty):
Triplanes for neural fields were first proposed in "Convolutional Occupancy Networks", Peng et al. and should be discussed.

Author response:
Updated draft: we have updated (in-place) the abstract, introduction, related work, conclusion sections. We’ve mentioned in the updated abstract that we used multi-view posed data for training. We’ve cited SRT paper and discussed the scope difference between our work and SRT. We also added citations to the Convolutional Occupancy Networks paper and FlowCam paper, as the reviewer suggested. For the experiment section, we did not modify it in-place because our modifications will change the figure/table numbers, hence increasing the reading difficulty for the reviewers. Instead, we put the updated experiment section in Appendix B. We plan to integrate it upon acceptance. We are also happy to integrate it now, if the reviewers don’t think this increases reading burden. You can find our novel-view PSNR in Tab. 14 and our novel-view renderings (along with ground-truth) in Fig. 9. We also include a detailed limitation section in Appendix C.

Reviewer comment (Novelty):
The paper "FlowCam" also incorporates 3D shape information during the camera pose prediction process and should be discussed.

Author response:
Updated draft: we have updated (in-place) the abstract, introduction, related work, conclusion sections. We’ve mentioned in the updated abstract that we used multi-view posed data for training. We’ve cited SRT paper and discussed the scope difference between our work and SRT. We also added citations to the Convolutional Occupancy Networks paper and FlowCam paper, as the reviewer suggested. For the experiment section, we did not modify it in-place because our modifications will change the figure/table numbers, hence increasing the reading difficulty for the reviewers. Instead, we put the updated experiment section in Appendix B. We plan to integrate it upon acceptance. We are also happy to integrate it now, if the reviewers don’t think this increases reading burden. You can find our novel-view PSNR in Tab. 14 and our novel-view renderings (along with ground-truth) in Fig. 9. We also include a detailed limitation section in Appendix C.

Reviewer comment (Experiments):
If the authors want to claim that their method outperforms prior pose-estimation methods, then they have to provide a fair benchmark and train these other methods on the same datasets and with a comparable level of compute as well, or at least provide a clear ablation study where they uncover that their model scales better with data than prior methods.

Author response:
We sincerely understand your concerns of fair comparison with baselines. We spent the rebuttal period scaling up the training of the baseline RelPose++ on the Objaverse dataset, and report the performance of the re-trained baseline in the following table. We can see that our method (trained on exactly the same data) still outperforms the retrained baseline by a wide margin, demonstrating the superiority of our proposed method. We also attempted to scale up the training of baseline FORGE, but its complex multi-stage training (6 stages) seems non-trivial to scale up in our trials. We leave it to future work to investigate the scalability issue of FORGE.  

---

OmniObject3D

| Method                                	| R. error | Acc.@15   | Acc.@30   | T. error  |
|-------------------------------------------|----------|-----------|-----------|-----------|
| RelPose++ (w/o bg, pretrained)        	| 69.22	| 0.070 	| 0.273 	| 0.712 	|
| RelPose++ (w/o bg, trained on Objaverse)  | 58.67	| 0.304 	| 0.482 	| 0.556 	|
| Ours (S)                              	| 15.06	| 0.695 	| 0.910 	| 0.162 	|
| Ours (L)                              	| **7.25** | **0.958** | **0.976** | **0.075** |

---

GSO

| Method                                	| R. error | Acc.@15  | Acc.@30  | T. error  |
|-------------------------------------------|----------|----------|----------|-----------|
| RelPose++ (w/o bg, pretrained)        	| 107.49   | 0.037	| 0.098	| 1.143 	|
| RelPose++ (w/o bg, trained on Objaverse)  | 45.58	| 0.600	| 0.686	| 0.407 	|
| Ours (S)                              	| 13.08	| 0.848	| 0.916	| 0.135 	|
| Ours (L)                              	| **2.46** | **0.976**| **0.985**| **0.026** |

---

ABO

| Method                                	| R. error  | Acc.@15   | Acc.@30   | T. error  |
|-------------------------------------------|-----------|-----------|-----------|-----------|
| RelPose++ (w/o bg, pretrained)        	| 102.30	| 0.060 	| 0.144 	| 1.103 	|
| RelPose++ (w/o bg, trained on Objaverse)  | 45.39 	| 0.693 	| 0.708 	| 0.395 	|
| Ours (S)                              	| 26.31 	| 0.785 	| 0.822 	| 0.249 	|
| Ours (L)                              	| **13.99** | **0.883** | **0.892** | **0.131** |

---

Reviewer comment (Evaluation):
In Figure 3 and Figure 4, I am confused about why the results are comparing input images with their predictions? Shouldn’t it be comparing GT novel view images with rendered novel views, as in Figure 5? Given input images, it’s not surprising that the reconstruction of input images should be almost perfect right?

Author response:
We thank the reviewer for the valuable feedback. We will add the suggested references ([a],[b]) in a revised draft soon before the rebuttal deadline. Please find our answers to your questions below. 

**Input-view PSNR**: We will include novel-view psnr in a revised draft before the rebuttal deadline.  The current input-view PSNR is to show that our predictions (both NeRF and poses) are well-aligned with the input images, as we render our predicted NeRF using our predicted poses; we would like to highlight that we do well in this important objective of matching re-renderings with inputs for joint 3D and pose prediction, while baselines struggle with this goal. We also show extracted geometry in Fig. 3, 4 and interactive NeRF viewers on our project page to demonstrate that we did not cheat by predicting degenerate solutions.

Reviewer comment (Novelty):
Missing related works. Section 3.3 is about pose predictions and PnP. This approach is closely related to the line of work in *scene coordinate regression* [a][b], which is not mentioned at all in the related work section.  

**Reference**

[a] Shotton, Jamie, Ben Glocker, Christopher Zach, Shahram Izadi, Antonio Criminisi, and Andrew Fitzgibbon. "Scene coordinate regression forests for camera relocalization in RGB-D images." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2930-2937. 2013.  

[b] Brachmann, Eric, Alexander Krull, Sebastian Nowozin, Jamie Shotton, Frank Michel, Stefan Gumhold, and Carsten Rother. "Dsac-differentiable ransac for camera localization." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6684-6692. 2017.

Author response:
We thank the reviewer for the valuable feedback. We will add the suggested references ([a],[b]) in a revised draft soon before the rebuttal deadline. Please find our answers to your questions below. 

**Input-view PSNR**: We will include novel-view psnr in a revised draft before the rebuttal deadline.  The current input-view PSNR is to show that our predictions (both NeRF and poses) are well-aligned with the input images, as we render our predicted NeRF using our predicted poses; we would like to highlight that we do well in this important objective of matching re-renderings with inputs for joint 3D and pose prediction, while baselines struggle with this goal. We also show extracted geometry in Fig. 3, 4 and interactive NeRF viewers on our project page to demonstrate that we did not cheat by predicting degenerate solutions.

Reviewer comment (Novelty):
How much of the improvement comes from the data? The pipeline is not that novel since all of which the authors have presented have been shown before, even the differentiable PnP solver comes from prior work. I would have liked to see scaling laws of the performance on the increased number of data points.

Author response:
**Limited pipeline novelty**: We respectfully disagree with this point. We would like to reiterate the contributions of our work: first, we propose a novel single-stream transformer architecture for joint NeRF and pose prediction, which is different from the encoder-decoder one used in the LRM, Instant3D work; the single-stream architecture not only allows the NeRF tokens to be contextualized by image tokens for reconstruction purpose, but  also allows the image tokens to be contextualized by NeRF tokens for predicting per-view coarse geometry used for solving poses. Second,  as far as we know, we are the first to apply the differentiable PnP solver to the task of jointly predicting 3D and pose from unposed sparse views, though the solver itself is inspired by the EPro-PNP work. This is in stark contrast to most prior work, e.g., RelPose++, Forge, that uses direct pose regression from images. Meanwhile, unlike the 6DoF pose estimation and 3D object detection tasks shown in EPro-PNP, we don’t have access to ground-truth 3D points/models during training (we only use multi-view posed images as training data); hence we propose the novel solution of supervising the sparse point predictions by distilling the predicted NeRF geometry in an online manner.

**Scaling laws**: we have shown scaling laws for model sizes in Table 1,2,3 and 4, where bigger models clearly outperform smaller models. Due to limited compute resources, we do not show scaling laws for data sizes using our largest model (as the scaling law of data is not insightful under smaller model and with inadequate compute [1]); but we’re happy to include one in the final draft.     [1] Scaling Laws for Neural Language Models

Reviewer comment (Experiments):
No comparison with sparse NeRF methods [1, 2, 3] is shown. Is this because they require accurate camera poses? I would have liked to see some comparison or discussion with these methods, some of them already show sparse novel view synthesis using triplane formulations [1].

Author response:
**Comparison with sparse NeRF methods**:  In this work, we focus on the challenging case of sparse unposed images as inputs, for the practical concern of estimating poses from sparse unposed images is a very challenging task itself. Hence, most sparse NeRF methods (e.g., [1,2,3] as the reviewer pointed out) requiring poses cannot work on such unposed inputs, especially when overlapping between input views is minimized (e.g., the frog and bunny in our Fig. 1).

Reviewer comment (Experiments):
Relating to point 1, how much does the network learn training data distribution? Is it possible that due to the massive scale of the objaverse dataset, some of the generalizable evaluation might be in question, since we don't know if the network has seen similar data before. Can the authors comment on that with some analysis/discussion?

Author response:
**Overfitting concern**:  As shown by Fig. 1, we have tested our method on the generated multi-views from Zero1-to-3, MVDream, SyncDreamer etc. These images are unlikely to be in the Objaverse dataset, as they are generated by 2D diffusion models, but our method still works well on them, showcasing our model’s strong cross-dataset generalizability. Moreover, we’ve also demonstrated our method’s generalization to the real scanned datasets OmniObjects3D, as shown in Fig. 3 and Tab. 1. This dataset was published later than Objaverse, making it unlikely to be covered by Objaverse.