Paper: An Analytical Solution to Gauss-Newton Loss for Direct Image Alignment (ICLR-2024)

Reviewer comment (Theory):
It would be interesting to see more discussions on the insight to the end-to-end learning framework's limitation, and a solution to that.

Author response:
Regarding the points raised in the weaknesses section, they indeed present interesting directions for further research. We are currently exploring the feasibility of end-to-end network training with our closed-form solution. In our preliminary experiments, we have found that backpropagation through this solution poses challenges due to numerical instability, and we are actively investigating potential solutions to this issue.

Reviewer comment (Experiments):
It would be interesting to see this approach handles outliers inherently.

Author response:
As we explain in the Implementation Details appendix, we currently address outliers only at the feature alignment stage. To manage outliers prior to the feature map creation stage (on the point of interest’s level), information from both reference and target images is required. However, our map creation process is based solely on a single image.

Reviewer comment (Experiments):
It would be interesting to see this approach is applied to other areas.

Author response:
We are also excited about the prospect of applying our method in other domains.

Reviewer comment (Experiments):
The paper is required to give more comparisons with state-of-the-art in terms of accuracy of SE3.

Author response:
In response to your first point about comparing our method with state-of-the-art methods in terms of SE3 accuracy, we have conducted extensive evaluations across various well-known datasets. Our comparisons, as detailed in the paper (e.g., in Table 1), focus on the Recall with respect to translation and rotation thresholds, which directly reflect the accuracy of the estimated SE(3) camera pose. If there are specific datasets or aspects that you believe we may have overlooked, we would appreciate your guidance to further enhance our comparative analysis.

Reviewer comment (Reproducibility):
Can the authors provide more training details of the proposed method, for example, the feature embedding network E, the learning rate, the batch size.

Author response:
Regarding your second point on the absence of training details, our method does not involve a traditional training process. However, it does leverage self-supervised SuperPoint descriptors. We will make it clearer in the updated manuscript.

Reviewer comment (Experiments):
My biggest gripe with the paper is that their claim that motivates the approach is not empirically validated and there is no mention of such validation elsewhere. The authors claim both in the abstract as well as in the appendix that prior art use feature maps that may embed the inductive bias of the training data. While I can comprehend the underlying reasoning, such a claim needs to be empirically shown. For example, an experiment on out-of-distribution test sets demonstrating the superiority of the closed-form solution would back the authors claims and in turn strengthen the paper.

Author response:
To empirically support our claim regarding the inductive bias of training data, we have added additional analysis to Appendix A.

Reviewer comment (Writing):
On the same topic of bias, I argue that the authors should explicitly state that their method still exhibits bias, but that the source of this is the underlying feature representation (result of this can be seen in Tbl. 1, Aachen Night dataset). Otherwise it may read that the authors claim their method is not biased. This is stated at the end of Section 3, but I think it should be stated clearly in either Abstract, Introduction and Conclusion section. This is a minor point however and only serves to improve clarity.

Author response:
We have refined our claim on the absence of bias in our method (at the end of Sec. 3) to make it more concrete.

Reviewer comment (Writing):
A little contradictory to my point in the Strengths section, I believe the math heavy section 4 and 5 could be made a little more clearer when derivations skip multiple steps. Otherwise the sections read as if one equation immediately follows from the other, which is not always the case. (e.g Eq 6. -> Eq. 7). This would enhance the readability of the paper.

Author response:
We modified the transition between Eq. 6 and Eq. 7 to show that there is no direct connection between them.

Reviewer comment (Experiments):
On the Aachen-Night dataset, the proposed method clearly suffers. The authors claim that this is due to the underlying feature representation used, which was not trained day-night correspondences. While I find the reasoning sound, it would help the authors claim to have used a feature representation that have used such correspondences during training. This in turn would again strengthen the papers contribution and indicate that it can work in different settings.

Author response:
We agree that using a feature representation trained on day-night correspondences would be a significant enhancement. Unfortunately, we could not find an off-the-shelf feature embedder trained for the day-night pairs.  
(Note that training such a feature embedder would require supervision.)

Reviewer comment (Experiments):
Fig. 4) The median is stable for all tested errors. For what ranges does this hold? I.e How far can the initial error be?

Author response:
The revised version of Fig. 4 now contains an extended analysis of the stability of the median error across broader ranges.

Reviewer comment (Presentation):
Fig. 4) Is there a similar plot for translation?

Author response:
We have added a corresponding plot for robustness w.r.t. translation to supplemental material.