Paper: Cameras as Rays: Pose Estimation via Ray Diffusion (ICLR-2024)

Reviewer comment (Theory):
Basically, the idea is to regress a ray represented as a 6D vector for each patch in the RGB image, which is arguably more challenging than predicting R and T. The difficulty lies in two aspects: first, it is a dense prediction problem; second, it regresses 3D information from RGB images. One could also predict the corresponding 2D coordinates in the right image for each patch in the left image as an alternative. Intuitively, it is easier to predict 2D coordinates than 6D ray vectors. The authors argue that such a method could struggle in sparse view settings due to insufficient image overlap to find correspondences, yet it is unclear why the presented method is able to achieve better robustness.

Author response:
We agree that predicting dense 2D correspondences between image pairs is an interesting alternative. However, converting such 2D correspondences to 3D cameras (in the presence of uncertainty, partial visibility, etc) is still challenging. Moreover, this is fundamentally a pairwise prediction task and is hard to extend to multiple images where the question of which images should be matched is also uncertain. Because our solution is not restricted to such pairwise reasoning, it bypasses this issue altogether. In addition, the representation inferred is directly convertible to global cameras.

We also note that our COLMAP baseline essentially does the sparser version of such a 2D correspondence-to-camera pipeline using a state-of-the-art feature matcher. Such methods lack robustness because of insufficient 2D correspondences.

Reviewer comment (Theory):
It is confusing why the presented method can recover the translation. According to Eq.3, m is coupled with the translation t. Predicting m then demands capturing information about the camera translation. However, the actual input of the network is a cropped image, meaning that the information regarding t is lost after the cropping.

Author response:
Thank you for pointing this out. We do provide the bounding box information of the crop to the network which is important for reasoning about translation (as the reviewer notes). Specifically, we concatenate the pixel coordinate of each patch (in normalized device coordinates with respect to the uncropped image) to the spatial features for both the ray regression and ray diffusion models that we train. 

We apologize for missing this detail in the earlier version and have now clarified this in the text and added the pixel coordinates to Eqs 7 and 11.

Reviewer comment (Experiments):
The authors only conducted experiments on the Co3D dataset, which makes the evaluation not convincing enough. There are several datasets that have been widely used in the literature such as Megadepth, ScanNet, and HPatches. It would be beneficial if the authors could show the effectiveness of the over parameterization on such datasets.

Author response:
We thank the reviewer for their response, and are glad to hear the misunderstanding was addressed and that the reviewer does appreciate the technical contributions of our work. However, as the reviewer still indicated they may lean towards rejection, we wish to state our case again before the discussion period ends. Based on the reviewer comments above, the primary concern that now affects their rating is that they believe it is critical to show empirical results on datasets like MegaDepth. While we appreciate the comments that we (and the field) should examine these setups, we feel that setting this new evaluation setup as a barrier for acceptance is an unreasonable (and counterproductive) bar for an emerging research area.

Specifically, learning-based methods for pose prediction with sparse-view/wide-baseline input represent a growing body of work, where prior methods (e.g. RelPose [ECCV 22], SparsePose [CVPR 23], PoseDiffusion [ICCV 23], RelPose++ [3DV 24]) have evaluated object-centric pose estimation in datasets like CO3D. In this commonly adopted setup, which is of practical significance in applications like reconstruction in online marketplaces, our approach clearly outperforms all these baselines. Moreover, following PoseDiffusion, we also demonstrate generalization to a scene-centric dataset RealEstate10k, where our approach again improves over prior work. Given these empirical results in the standard evaluation setup adopted by prior work, combined with the technical contributions (that the reviewer agrees on), we believe that our work clearly represents a step forward for this area of learning-based pose prediction and that the community would benefit from it.

Again, we understand the reviewer may wish to see methods such as ours being evaluated in other scenarios that are commonly studied in the SfM community. But such change occurs gradually, and should not be enforced overnight! For example, PoseDiffusion proposed a scene-centric generalization, and our work now incorporates the preliminary results on MegaDepth (which will hopefully encourage others to follow suit). We would therefore urge the reviewer to reconsider, and respectfully, not enforce a threshold that deviates from common practice in prior works as a criterion against acceptance of our work.

Reviewer comment (Experiments):
According to Eq.7, the patches of all available images are jointly processed, which is computationally expensive. As reported by the authors, training the diffusion model takes four days on 8 A6000 GPUs, which is much slower than RelPose and RelPose++.

Author response:
The per-iteration training speed of our method is similar to that of RelPose++, but we train for half the iterations, so our total training time is about half that of RelPose++ (4 days vs 8 days of training on 8 A6000s). As we report in Table 6, the inference speed of our Ray Diffusion method is 2.6X faster than RelPose while our Ray Regression method (which requires a single forward pass) is 220X faster.

Reviewer comment (Theory):
Most of the equations in this paper make sense to me, but the explanation of Eq.5 is a bit confusing. What is the “identity” camera? Are there any constraints on this equation? Does it still hold when the image depicts multiple planes?

Author response:
By identity camera, we refer to a camera for which both the intrinsics and rotation are identity matrices: $K=I, R=I$. We have now clarified this in the text.

If we have a camera with rotation $R$ and intrinsics $K$, the direction of the ray corresponding to a (homogenous) pixel coordinate $u_i \in \mathbf{P}^2$ can be computed as $R^T K^{-1} u_i$. Given predicted directions $d_i$, we thus need to compute optimal $R, K$ such that $R^T K^{-1} u_i = d_i$,  or equivalently, $(K R) d_i = P d_i = u_i$. Note that this is an ‘equality’ in homogenous representations (up-to-scale). Solving this corresponds to finding the optimal homography matrix such that $ H d_i = u_i$.

Our terminology stems from an alternate interpretation where ‘directions’ can be thought of as points on the plane at infinity (in projective 3D space) i.e. a direction $d_i  \in \mathbf{P}^2$ implies a point $(d_i,0) \in \mathbf{P}^3$. In this interpretation, finding $K,R$ is equivalent to finding the homography that relates  the images of this plane at infinity under the two cameras: identity camera $(K=I, R=I)$ and the regular camera $(K=K, R=R)$ (see Sec 8.5 in Hartley & Zisserman). 

Please note that this does not have anything to do with planar surfaces present in the scene; and eq 5 holds for any scene geometry.

Reviewer comment (Evaluation):
As shown in Table 1, sometimes the performance of the presented method decreases when more images are involved. By contrast, the performance of most competitors such as RelPose++ consistently becomes better with more images. I was wondering why the method is not compatible with multi-view images.

Author response:
This issue appears to be due to a bug when computing our camera intrinsics at training time (see top-level comment). Now that the bug has been fixed, the performance does not decrease with more images as expected, as shown in Table 1. We apologize for the confusion.

Reviewer comment (Writing):
The punctuation is necessary at the end of each equation, please check it carefully.

Author response:
Thank you for these suggestions. We have added punctuation to the end of equations and updated the reference format.

Reviewer comment (Reproducibility):
The authors fail to state more details of the proposed network architecture. Moreover the training detail is also required.

Author response:
We use standard network architectures: DINOv2 for feature extraction and DiT for regressing and diffusing camera rays. We have included additional training details such as dataset preparation, model architecture, and diffusion hyperparameters in Section 3.4 Implementation Details. We would be happy to clarify any other details, and we will be releasing code to ensure reproducibility.

Reviewer comment (Experiments):
To demonstrate the performance of the proposed novel representation, can authors undertake more experiments on more datasets?

Author response:
We have added experiments on zero-shot generalization to RealEstate10K (following PoseDiffusion) as well as newly trained results on MegaDepth (see top-level comment). We find that our method has significantly better zero-shot generalization compared to previous methods in this scene-centric setup.

Reviewer comment (Presentation):
Please check the format of REFERENCES.

Author response:
Thank you for these suggestions. We have added punctuation to the end of equations and updated the reference format.

Reviewer comment (Experiments):
One dataset is too small to see the applicability of a method. Since I see this method as superior to "PoseDiffusion", it would be great to see some results on the "scene-centric" dataset and compare it against PoseDiffusion.

Author response:
Please refer to the top-level response. We have added experiments on zero-shot generalization to RealEstate10K (following PoseDiffusion) as well as newly trained results on MegaDepth.

Reviewer comment (Reproducibility):
It would be nice to see a "memory" requirement to run these models. Processing N image features together, I am assuming requires a good amount of GPU memory.

Author response:
We thank the reviewer for this suggestion. We have added a GPU memory analysis for our Ray Diffusion model to the paper in Table 7. According to `nvidia-smi`, backward diffusion using our ray diffusion model on 8 images reaches a peak GPU memory usage of 3095 MiB of VRAM, which should fit on any standard GPU.

Reviewer comment (Evaluation):
It would also be nice to see accuracy at different thresholds i.e. @5, @10, @15.

Author response:
In Tables 10 and 11, we analyze the rotation and camera center accuracies at a variety of thresholds. We have also added a new measure of rotation and camera center using AUC which evaluates accuracies at all thresholds in Tables 8 and 9, and visualize the threshold vs accuracy curve in Figure 11.

Reviewer comment (Experiments):
It would also be nice to see an ablation study where we do not scale the poses. Most of the applications require properly "scaled" poses.

Author response:
We would like the reviewer to clarify what is meant by not scaling poses. We are interpreting the reviewer’s question as whether it’s possible to not normalize camera scale at training time and instead learn metric scale. In our camera normalization procedure, the sampled minibatches of cameras are re-scaled such that the first camera always has a unit translation. This procedure is done because the ground truth camera poses are acquired using COLMAP which can produce scenes in an arbitrary scene scale, which is not consistent between sequences. Thus, it is not possible to learn metric scale since the magnitude of translations would be ill-defined. If we have incorrectly interpreted the reviewer’s statement, we politely request clarification.

Reviewer comment (Presentation):
The language is clear but I think the paper presentation is poor.

Author response:
We thank the reviewer for the feedback and have made revisions accordingly.
We have re-made all the qualitative results figures (Figure 5 onward). We respectfully ask the reviewer for feedback on the new presentation of results.
For Figure 2 specifically, we are trying to convey 1) that the camera and ray representation are easily interchangeable and 2) that even noisy rays (e.g. if they do not intersect at a single point) can still be converted into a valid camera. We would appreciate any feedback from the reviewer on how to make this more clear. We have updated the caption to add more context.

Reviewer comment (Presentation):
Fig 2 is really confusing where the authors are trying to show the camera to ray-bundle and ray-bundle to camera process.

Author response:
For Figure 2 specifically, we are trying to convey 1) that the camera and ray representation are easily interchangeable and 2) that even noisy rays (e.g. if they do not intersect at a single point) can still be converted into a valid camera. We would appreciate any feedback from the reviewer on how to make this more clear. We have updated the caption to add more context.

Reviewer comment (Presentation):
Fig 5 — A qualitative comparison is hard to see and to make a good sense of, as opposed to Fig 4 of PoseDiffusion paper for example.

Author response:
We have re-made all the qualitative results figures (Figure 5 onward). We respectfully ask the reviewer for feedback on the new presentation of results.

Reviewer comment (Experiments):
I see runtime in the appendix, but how does this method scale with adding a number of views? Instead of 8 what if I had 32? What are the memory requirements and inference runtime graphs?

Author response:
Although our model is trained with 8 images, we find that it can generalize effectively to more images by re-sampling mini-batches during the backward diffusion process. Specifically, at each iteration of DDPM, we can sample batches of 8 images (keeping the first image fixed), and predict the $X_0$s for each batch separately. With more images, the memory cost is constant but the runtime scales linearly. We find that our performance remains consistent with more images despite training with only 8 images, as we report below and added to Table 5 in the paper

| # of Images                           | 8    | 15   | 22   | 29   | 36   | 43   |
|----------------------------------------|------|------|------|------|------|------|
| Rotation Acc. (Seen Categories)        | 93.3 | 93.1 | 93.3 | 93.1 | 93.4 | 93.4 |
| Rotation Acc. (Unseen Categories)      | 88.1 | 88.2 | 89.2 | 88.7 | 89.0 | 88.9 |
| Camera Center Acc. (Seen Categories)   | 84.1 | 78.3 | 76.5 | 75.3 | 74.7 | 74.2 |
| Camera Center Acc. (Unseen Categories) | 71.4 | 62.7 | 61.1 | 59.3 | 59.2 | 58.9 |

Reviewer comment (Experiments):
Authors say they stopped the backward diffusion process early and found that those estimates were better. How early did they stop the diffusion? And how did they choose when to stop? It would be nice to see some ablation analysis.

Author response:
In our initial experiments, we found that our accuracy metrics generally peak around T=30 (where T=100 is complete noise and T=0 is the “normal” final diffusion timestep). We have added an ablation that shows performance at all timesteps in Figure 12.