Paper: AutoNeRF: Training Implicit Scene Representations with Autonomous Agents (ICLR-2024)

Reviewer comment (Evaluation):
How the automatic process helps the downstream tasks is unclear. It is known that a trained NeRF can be applied for these four tasks (as also mentioned in the Related Work section). The paper directly adopts the methods without further modification, and the NeRF training does not provide any feedback to the exploration policy training. Moreover, the exploration phase already builds a 2D map online that can be used for Task 2 & 3. The benefit of involving an extra NeRF training phase for these two tasks should be further elaborated.

Author response:
Downstream tasks are used in this work as an evaluation protocol of the underlying quality of a given NeRF model. While many recent works focus on evaluating NeRFs through novel view synthesis, we consider a suite of tasks that allow a more thorough evaluation. Among these tasks is occupancy map estimation from the NeRF representation which allows to verify the geometric understanding of the 3D scene by the NeRF model.

Reviewer comment (Experiments):
What is the role of semantic reasoning? Seemingly the semantic cues are not the best reward for exploration, and semantics are not a prerequisite for the 'AutoNeRF' setting. The subtasks of semantic rendering and object-goal navigation can be removed without hurting the central contribution. The role of semantics should be better demonstrated.

Author response:
Semantics is closely tied to geometry. Properly covering the whole scene with diverse viewpoints and camera positions leads to a better geometric representation of a 3D scene. Being able to map semantic information requires a good knowledge of geometry, which makes semantic rendering a relevant task to evaluate NeRF quality. Moreover, semantic mapping is important from an applicative point of view if we consider the final NeRF model as a persistent representation of a scene of interest.

Reviewer comment (Presentation):
Does the training of NeRF lead to a better BEV map? The exploration phase already obtains a 2D metric-semantic map on the fly that can be applied for exploration and point/object-goal navigation. The map should be added to Fig. 10 for a better understanding of how the performance can be affected with a follow-up NeRF model.

Author response:
As already mentioned in a previous part of our answer, downstream tasks are used in this work as an evaluation protocol of the underlying quality of a given NeRF model. The main point of evaluating the BEV map derived from the NeRF model is to assess the quality of its geometric understanding.

Reviewer comment (Novelty):
The novelty of the paper is limited. The idea of using Nerf for scene construction in SLAM is not new.

Author response:
> The novelty of the paper is limited. The idea of using Nerf for scene construction in SLAM is not new. The authors also adopted an off-the-shelf NERF module. The main contribution of this paper lies in validating the effect of different exploration policies during image collection on downstream robotics tasks.

Please refer to our global answer to all reviewers for a detailed description of the contributions of this paper.

Reviewer comment (Novelty):
The authors also adopted an off-the-shelf NERF module.

Author response:
> The novelty of the paper is limited. The idea of using Nerf for scene construction in SLAM is not new. The authors also adopted an off-the-shelf NERF module. The main contribution of this paper lies in validating the effect of different exploration policies during image collection on downstream robotics tasks.

Please refer to our global answer to all reviewers for a detailed description of the contributions of this paper.

Reviewer comment (Reproducibility):
How many images are used to train the NERF model? In task specification it says "The agent can navigate for a limited number of 1500 discrete steps", does it mean to capture 1500 images for NERF training?

Author response:
> How many images are used to train the NERF model? In task specification it says "The agent can navigate for a limited number of 1500 discrete steps", does it mean to capture 1500 images for NERF training?

The agent can indeed navigate for 1500 steps, leading to a training set that is made of up to 1500 training samples. However, we perform a filtering to remove images associated with close camera poses. In practice, when deploying the modular policy trained with the *obstacle coverage reward* on previously unseen scenes, the mean number of collected images is 754, with a minimum of 264 in the smallest scene and a maximum of 1257 in the largest scene.

Reviewer comment (Experiments):
Could you please share the results of the vanilla Semantic NeRF model mentioned in Figures 9 and 10? While it is claimed to deliver superior results compared to Semantic Nerfacto, there is a lack of supporting evidence.

Author response:
The *vanilla Semantic NeRF* model leads to a higher-quality 3D mesh after applying our mesh extraction protocol. The difference was assessed qualitatively by witnessing holes and irregularities in the *Semantic Nerfacto* final mesh. The latter still provides satisfying results as can be seen quantitatively and with qualitative BEV maps, and is much faster to train, but our aim here was to show the best practical results in terms of final 3D scans one could get when applying AutoNeRF.

Reviewer comment (Evaluation):
It appears that the four types of rewards exhibit different strengths. Are there any criteria or trends that can assist researchers in selecting the most appropriate reward type for specific tasks? For instance, is there a reason why the policy with the obstacle coverage reward performs better than others in the rendering task?

Author response:
Different reward functions can lead to diverse policy behaviors, which is why we report the performance of the different modular policies on all the downstream tasks. The chosen reward might depend on the specific considered task, but in this work we were rather interested in finding the reward function that leads to the best overall performance on the different downstream tasks. From collected results, obstacle coverage and viewpoints coverage lead to the best overall performance.

In this work, the rendering ability of a NeRF model is evaluated by sampling camera poses in the scene and comparing NeRF rendering with rendering from the simulator. The difficulty of the task is thus linked to rendering different objects that might not have been observed thoroughly, while redundant parts of the scene such as floors, walls, ceilings have likely been observed from different viewpoints by any policy with ease. This explains the high performance of the obstacle coverage reward, as it guides the policy to focus on covering specific objects or obstacles in the scene.

Reviewer comment (Evaluation):
Could you provide a more detailed explanation of the experiments featured in Table 1 and elaborate on their implications?

Author response:
Table 1 presents an application of AutoNeRF to a practical scenario where a user would have bought a brand new robot that navigates using a neural policy (different from AutoNeRF) that was pre-trained in simulation. They now want to use the robot in their house, but even if the robot policy was pre-trained on a large and diverse set of scenes, it is quite likely it will struggle with some specificities of their house. A solution to this would be scene-specific fine-tuning: fine-tuning the robot policy on their house. However, such fine-tuning must be performed safely so a good thing would be to do it in simulation. A question is thus: can we use AutoNeRF to autonomously explore a new scene, collect data and train a NeRF? We can then extract a mesh representation from the trained NeRF, load this mesh into a simulator and fine-tune the robot policy of interest. Table 1 shows results of similar experiments performed in simulation: we show that such a procedure allows to improve the performance of a PointGoal agent, pre-trained on diverse scenes, by fine-tuning it on previously unseen scenes scanned by AutoNeRF.

Reviewer comment (Novelty):
The scientific contribution of this paper is unclear. Although the authors conduct a comprehensive evaluation of the designed autonomous approaches, it does not point out their core contributions to this field. The proposed autonomous exploration strategy is straight-forward, and could not be considered as main contribution of this paper. Additionally, the absence of a comparison between their methods and existing techniques in autonomous visual exploration—which could have been utilized for collecting NeRF training data—is missing. The experimental evidence alone does not constitute significant scientific contribution.

Author response:
Please refer to our global answer to all reviewers (“Contributions of this work” section) for a detailed description of the contributions of this paper. As mentioned, our contributions are on the experimental side, and we do compare the introduced modular policies with state-of-the art end-to-end visual exploration methods (*Ramakrishnan et al. An exploration of embodied visual exploration, IJCV 2021*). We believe that autonomous NeRF training data collection is an important topic and, to the best of our knowledge, this is the first work to target this direction, providing diverse experimental results.

Reviewer comment (Experiments):
Did you evaluate the end-to-end based and modularized based method on completed unseen (or largely different) scenario? I am curious about the actual performance in such scenarios. While the rule-based method, i.e., the frontier-based exploration, did not outperform learning-based methods (such as end-to-end based and modularized methods) in your experiments, it demonstrated fairly consistent results across the majority of scenes, regardless of whether they were familiar or unfamiliar scenarios.

Author response:
Please refer to our global answer to all reviewers (“Generalization to new scenarios” section) for a description of our experimental setup which applies to end-to-end and modular methods.