Paper: End-to-End (Instance)-Image Goal Navigation through Correspondence as an Emergent Phenomenon (ICLR-2024)

Reviewer comment (Experiments):
It is not clear on which databases or environments the system has been tested. For example, on what environments are the results reflected in Table 5 obtained? The interested reader has to review (Krantz et al., 2023) to know these details. Section 4, "Experimental setup" subsection needs to be improved.

Author response:
Instance Imagenav task has been trained and tested on the Habitat-Matterport3D (HM3D) Semantics v0.2 2 dataset that was provided for the Habitat Navigation Challenge 2023. The dataset contains 216 different scenes; the train/val/test split is 145/36/35. Our model was trained on the 145 training scenes, we reported results for the 36 validation scenes. The remaining 35 test scenes are used by organizers internally; they are not publicly available. 

This information has been added to the appendix of the paper.

Reviewer comment (Reproducibility):
One of the most interesting contributions of the proposed model is that it generates correspondences between images due to how the pre-training is designed. Figure 4 shows some results or qualitative evidence in this regard. However, the paper does not explain in detail how this image is generated or how these correspondences are analyzed. The manuscript simply mentions that they "visualize averaged attention of the last cross-attention layer of a DEBiT-L model", but more details should be provided.

Author response:
The visualization is simple:  we have examined the last cross-attention layers, with attention summed over all heads, and looked at individual attention values. We have then picked the highest N attention values in this matrix and displayed their corresponding query-key pairs, i.e., drawing a line between the corresponding patches.

This has been added to the paper and to the video.

Reviewer comment (Presentation):
The limitation and future extension of this work are not discussed in this paper.

Author response:
> The limitation and future extension of this work are not discussed in this paper.

Limitations: 
* further improvements could be done on the Instance-ImageNav benchmark by changing the pre-training tasks such that image pairs are also sampled with different camera intrinsics, as currently this difference is only learned through the RL loss and with adapters.
* increasing the image resolution might be beneficial.

Future and ongoing work will address the following aspects:
* using DEBiT for visual odometry, 
* Pre-training on frame pairs which correspond to the Instance-ImageNav setting 
* Training the model on pairs where the background of the goal image is segmented out by a model like “Segment Anything”
* Porting the model to our navigation software on our real robotics platform

This has been added to the paper.

Reviewer comment (Presentation):
The title is somewhat misleading as the emergence of correspondences is really not the focus of the work, but more of an afterthought. The content of the paper might be mistaken as an investigation into this phenomenon that is carried out in Tang et al, Emergent of Correspondences from Image Diffusion, arXiv, 2023.

Author response:
While the emergence of correspondence is not the focus of this work, we nevertheless think that it is an interesting finding, in particular when compared to competing methods for this problem, which either calculate correspondences explicitly (Krantz et al, 2023), or are based on late fusion, where correspondences is hard or impossible to emerge. We therefore argue that this sets our work apart from the literature on this topic.

Thank you for the references, we have added them to the paper.

Reviewer comment (Novelty):
The emergence of correspondences from the learned representation is not really that surprising, given that the pre-training task is relative pose estimation. Even before the introduction of transformers, monocular pose estimation methods have shown that the representation learns to identify meaningful keypoints on objects. One example: Mousavian et al, 3D Bounding box estimation using deep learning and geometry, CVPR 2017.

Author response:
While the emergence of correspondence is not the focus of this work, we nevertheless think that it is an interesting finding, in particular when compared to competing methods for this problem, which either calculate correspondences explicitly (Krantz et al, 2023), or are based on late fusion, where correspondences is hard or impossible to emerge. We therefore argue that this sets our work apart from the literature on this topic.

Thank you for the references, we have added them to the paper.

Reviewer comment (Experiments):
Although the inclusion of the experiment where DEBiT was integrated with ANS for a direct comparison to a modular approach is appreciated, the large performance gap to the proposed method is surprising. It seems unintuitive that a method that tries to map pixels directly to actions would outperform an approach that uses a map and de-couples the planning from the control. This should be examined more carefully to ensure fair comparison.

Author response:
ANS was not finetuned, we took the weights from the model provided by the authors. This will also explain a drop in performance. As said in the paper, this experiment was proof-of-concept of the possible integration of DEBiT into a modular method. It should not be seen as a direct comparison. We have toned down the language on the comparison in the paper.

We conjecture that a large gain is obtained by the fact that the end-to-end trained policy benefits from the richer latent embeddings passed from the pen-ultimate layers of the visual encoders, whereas ANS only receives pose and visibility directly. We have run an experiment to confirm this where  the end-to-end trained agent only receives pose and visibility. It reached only 20\% SR in training, which provides evidence that the full embedding is quite important for the end-to-end trained agent. We conjecture that the performance could be further increased by longer training, but we made it comparable and trained the agent for 200M steps.

We have added some elements of this to the paper.

Reviewer comment (Experiments):
It is unclear whether the global policy of ANS was fine-tuned with the frozen binocular encoder b and using the AdaptFormers. Since ANS was trained for object-goal, the Neural SLAM and global policy components might need to be re-trained for the ImageGoal task. As it stands, the experiment is not convincing that end-to-end methods are better than modular approaches.

Author response:
ANS was not finetuned, we took the weights from the model provided by the authors. This will also explain a drop in performance. As said in the paper, this experiment was proof-of-concept of the possible integration of DEBiT into a modular method. It should not be seen as a direct comparison. We have toned down the language on the comparison in the paper.

We conjecture that a large gain is obtained by the fact that the end-to-end trained policy benefits from the richer latent embeddings passed from the pen-ultimate layers of the visual encoders, whereas ANS only receives pose and visibility directly. We have run an experiment to confirm this where  the end-to-end trained agent only receives pose and visibility. It reached only 20\% SR in training, which provides evidence that the full embedding is quite important for the end-to-end trained agent. We conjecture that the performance could be further increased by longer training, but we made it comparable and trained the agent for 200M steps.

We have added some elements of this to the paper.

Reviewer comment (Experiments):
Why is using panoramas treated as unrealistic with regards to robotic applications? RGB sensors (and even RGB-D) are relatively cheap and can be easily mounted on robots to cover a 360 degree field-of-view.

Author response:
In the panoramic setting, both observed and goal images are supposed to be panoramic, and this is why the setting is not realistic. While 4 cameras can be simply put on a robot, the panoramic setting also requires that 4 different *goal* images are captured by user, which is often done with a hand-held camera or a cell phone, and it is required to do this with the same optical center and with exact angles of 90°. This is very complicated to do for an end user.

Reviewer comment (Novelty):
The authors cover some of the literature on pre-text tasks for improving sample efficiency, but additional works such as Ye et al, Auxiliary Tasks and Exploration Enable ObjectGoal Navigation, ICCV 2021, and Sax et al, Mid-Level Visual Representations Improve Generalization and Sample Efficiency for Learning Visuomotor Policies, CoRL 2019, are also worth discussing.

Author response:
Thank you for these references, we will add them to the paper. We indeed think that certain / several pre-text tasks are complementary to ours, and it would also certainly be worth exploring whether these tasks should be used to train the monocular encoder m, or whether the binocular encoder should take over parts of the reasoning they add.

Reviewer comment (Experiments):
Especially regarding Sax et al (2019), shouldn't a representation trained on a multitude of vision tasks be more task-generalizable compared to just pre-training on relative pose estimation?

Author response:
Thank you for these references, we will add them to the paper. We indeed think that certain / several pre-text tasks are complementary to ours, and it would also certainly be worth exploring whether these tasks should be used to train the monocular encoder m, or whether the binocular encoder should take over parts of the reasoning they add.

Reviewer comment (Presentation):
The last portion of supplementary video in terms of correspondence needs better representation and also that is the core focus.

Author response:
We have added an explanation to the video.

In essence, the visualization is simple: we have examined the last cross-attention layers, with attention summed over all heads, and looked at individual attention values. We have then picked the highest N attention values in this matrix and displayed their corresponding query-key pairs, i.e., drawing a line between the corresponding patches.

Reviewer comment (Evaluation):
The analysis of time complexity for doing correspondences in the benchmark and the distribution of work load should have helped understand the bottlenecks for a near real time system like robotic agents, even in embodied setups.

Author response:
Computational complexity is dominated by the visual encoder, and our training frame-rates indeed dropped when adding the attention based binocular encoder. However, we still get high frames rates in training on an A100:

* DEBiT-L : 92 fps
* DEBiT-B : 156 fps
* DEBiT-S : 192 fps
* DEBiT-T : 225 fps

This includes all forward passes over visual encoders and policy, and also things not needed in inference on a robot: backward passes and the simulation of 12 different Habitat environments in parallel. We did not yet test DEBiT on our own robots, but we have tested similar policies running on an embedded Nvidia Jetson and we can achieve decisions with delays of well under 100ms, which allows the robot to move quickly with 1m/s if dynamics if the delay is handled correctly in simulation as well.

In terms of complexity of the correspondences themselves, it is the complexity of attention, which is quadratic in terms of tokens (which are patches) and linear in the embedding dimension, number of heads and number of layers. We have input images of size 112x112  and patch sizes of 16x16 which gives 7x7 = 49 patches per image.

This has been added to the appendix of the revised paper.

Reviewer comment (Experiments):
A practical deployment in robotic setup should have confirmed the real world transfer applicability.

Author response:
We leave this for future work, integration in our real robotics platform will actually start very soon.

Reviewer comment (Presentation):
I think Fig. 1 image you have search to related the chair with big picture - can any other image or better resolution be used?

Author response:
We have zoomed into the observation in the revised paper, this should make the chair better visible.

Reviewer comment (Presentation):
Same with panoramic and mono view - please help in making sense of the image if at all included in body. If space constraint, appendix referral is there for later sections, but introduction setup has to be clear.

Author response:
We have zoomed into the observation in the revised paper, this should make the chair better visible.