Paper: NeRFuser: Diffusion Guided Multi-Task 3D Policy Learning (ICLR-2024)

Reviewer comment (Theory):
the reliance on point cloud voxelization means that the entire scene must be completely observed up front and small enough to be voxelized. this makes it unusable for e.g. autonomous driving / robot navigation in large scenes.

Author response:
Thanks for pointing this out. We agree this is a main challenge for voxel-based methods in large scenes. However, there are several ways to solve this problem. Firstly, our method is quite flexible and can be extended to using a point-based network to process point cloud directly and construct a 3D feature volume from which we sample point features for neural rendering and policy learning. This helps avoid point cloud voxelization and can be more efficient in large scenes, which is a promising research direction. In addition, previous work [1] has proposed to use occupancy networks to drastically reduce the memory requirements and increase the voxel grid resolution. This enables us to use a high-resolution voxel grid and effectively accommodate larger scenes.

Reviewer comment (Evaluation):
I am skeptical that the neural rendering objective can achieve stronger forms of generalization. "New position", "larger object" and "distractors" are all fairly weak forms of generalization, and I suspect the method would not be able to generalize to new environments or objects that have the same semantics but significantly different appearance from the training set. Ideally Stable Diffusion can help bridge this gap, but I assume the authors tried tougher OOD tasks and the method didn't work.

Author response:
Thank you for your question. Generalizing to novel objects or those with significantly different appearances can be challenging when training is limited to our existing datasets. However, the NeRFuser's design, which separates 3D semantic training from policy learning, offers the potential to train from large-scale multi-view datasets and achieve high generalizability as foundation models. This is because previous works require the datasets to contain both actions and multi-view information and only a scarcity of data can meet the requirements. On the contrary, in NeRFuser, the datasets for neural rendering and policy learning are necessarily required to be the same, which allows us to use a multi-view dataset without actions (e.g. SUN3D [1]) and a demonstration dataset that is not multi-view (e.g. RT-X dataset [2]) to train the NeRFuser.

Reviewer comment (Experiments):
going from a voxel grid to its radiance field is a little strange. the voxel grid already stores RGB and occupancy, so the 3D encoder basically has to learn the identity function and only recover view dependence to turn it into a radiance field. so I suspect the NeRF (RGB) objective barely contributes to learning at all. maybe the authors can verify / refute this.

Author response:
We have done experiments training NeRF without RGB objective during the pre-training phase. We found the RGB objective significantly affected the stability of the NeRF Pre-training process. Only using feature embedding objective without RGB can easily lead to model collapse in the early training stage.

Reviewer comment (Experiments):
the method seems quite general but the types of environments/tasks explored in this paper are fairly constrained. it would have been nice to see some more diverse environments / agents.

Author response:
Thank you for your suggestion. For simulation, we adopt RLBench, which contains a standard simulation environment and benchmarked tasks that are used by previous work for fair comparison and we follow the same setting to show that NeRFuser outperforms baseline significantly. For real robot experiments, we include 5 challenging real-robot tasks to evaluate the performance of the NeRFuser and the baselines. The number is greater than the 3 real-robot tasks used in the GNFactor.

Reviewer comment (Presentation):
I don't particularly like the name NeRFuser because it sounds like a method for improving/fusing NeRFs. I also feel like the neural rendering component here is not the most important part of the pipeline, and the 2D teacher is at least equally as important. See if you can come up with a better name.

Author response:
Thank you for pointing this out. Yes, we agree that there are multiple components that contribute to the flexibility and efficiency of this method. We have updated the name to DNAct, where D refers to Diffusion, N refers to NeRFs, and Act refers to action and policy.

Reviewer comment (Experiments):
I am not entirely sure what is the pre-training / training / test set for the real robot experiment. I think it would be interesting if you trained only on RLBench and tested on real robot tasks, but my understanding is that this combination was not tried. it would also be nice to repeat the real robot results several times to report stdev in the table.

Author response:
For real robot experiments, we use three RealSense cameras to collect a multi-view dataset for the pre-training phase and we simultaneously collect the demonstration dataset (for the following policy learning phase) with the front RealSense camera. To evaluate the model, we do not require an additional dataset and directory to obtain action predicted by the policy MLP.
It is indeed an interesting direction to utilize the simulation datasets to pre-train the model. In fact, one of the contributions of NeRFuser is that it separates the neural rendering and the policy learning components, which allows the model to have the flexibility to train and benefit from two distinct datasets such as SUN3D [2] multi-view dataset for indoor scene understanding and RT-X large-scale trajectory dataset [3] for policy learning. For the real robot experiments, it is still very time-consuming and expensive to repeat evaluations multiple times. In future work, we hope to explore more efficient methods that might allow for such repetitions.

Reviewer comment (Reproducibility):
can you describe exactly how the stable diffusion features are extracted? you feed in the input image, encode it, add noise (to which timestep?), then denoise (in 1 step)?

Author response:
Yes, we use RGB images and corresponding language instructions as input for the Stable Diffusion model. Specifically, we encode the language instruction with a pre-trained text encoder and extract the text-to-image diffusion UNet’s internal features by feeding language embedding and RGB images into the UNet without adding noise. The rich and dense diffusion features are then extracted from the last layer of the UNet blocks. We only require a single forward pass to extract visual features instead of going through the entire multi-step generative diffusion process.

Reviewer comment (Novelty):
The stated technical contributions have been proposed by previous works. In the abstract the work claims as a contribution: “NeRFuser leverages neural rendering to distill 2D semantic features from foundation models.” However, this exact setup was proposed by GNFactor [1].

Author response:
We appreciate your concerns about the perceived similarities between our work and previous studies. To clarify, our approach differs from GNFactor and other related methods in two primary aspects:

* Training: Unlike GNFactor, which employs joint training of NeRF and behavior cloning policy, our method uses pre-training with neural rendering. This distinct approach allows for better generalization and improved quality in neural rendering. While GNFactor’s joint optimization can lead to a conflict between objectives, affecting both rendering quality and semantic information utilization, our pre-training strategy avoids this issue. Our results demonstrate superior performance in multi-task robot learning compared to the Learning-from-Scratch baseline.

* Data Utilization: GNFactor requires task-specific, multi-view images paired with expert demonstration data, which can be challenging to collect and scale. In contrast, our approach effectively leverages widely available, task-agnostic multi-view image data. This not only eases the data collection process but also enhances the versatility of our 3D foundation model, making it applicable to a broader range of downstream robotic tasks. Our method's efficacy is further demonstrated through its application to five previously unseen tasks, yielding better multi-task performance compared to baseline PerAct and GNFactor.

Furthermore, it's important to highlight our incorporation of diffusion-based techniques for multi-modal representation. This aspect of our work marks a significant advancement over previous methods. We're one of the first to successfully apply diffusion training in multi-task manipulation, whereas earlier works, such as those by Janner et al. and Huang et al., are predominantly evaluated under single-task scenarios. By formulating representation learning as an action sequence reconstruction problem and integrating diffusion models, we achieve more stable training and eliminate the need for the time-consuming inference processes typical of these models. This multi-modal approach not only enhances our method's capabilities but also sets our work apart in the field of robotic task learning.
In summary, our approach stands out due to its unique pre-training approach, effective use of multi-view data, and innovative application of diffusion models for multi-modal representation, all contributing to its enhanced generalization and performance in diverse robotic tasks.

Reviewer comment (Novelty):
The other contribution is stated to be: "We introduce a novel approach utilizing diffusion training to learn a vision and language feature". This approach seems very similar to previous works Janner et al. [2] and Huang et al. [3].

Author response:
We appreciate your concerns about the perceived similarities between our work and previous studies. To clarify, our approach differs from GNFactor and other related methods in two primary aspects:

* Training: Unlike GNFactor, which employs joint training of NeRF and behavior cloning policy, our method uses pre-training with neural rendering. This distinct approach allows for better generalization and improved quality in neural rendering. While GNFactor’s joint optimization can lead to a conflict between objectives, affecting both rendering quality and semantic information utilization, our pre-training strategy avoids this issue. Our results demonstrate superior performance in multi-task robot learning compared to the Learning-from-Scratch baseline.

* Data Utilization: GNFactor requires task-specific, multi-view images paired with expert demonstration data, which can be challenging to collect and scale. In contrast, our approach effectively leverages widely available, task-agnostic multi-view image data. This not only eases the data collection process but also enhances the versatility of our 3D foundation model, making it applicable to a broader range of downstream robotic tasks. Our method's efficacy is further demonstrated through its application to five previously unseen tasks, yielding better multi-task performance compared to baseline PerAct and GNFactor.

Furthermore, it's important to highlight our incorporation of diffusion-based techniques for multi-modal representation. This aspect of our work marks a significant advancement over previous methods. We're one of the first to successfully apply diffusion training in multi-task manipulation, whereas earlier works, such as those by Janner et al. and Huang et al., are predominantly evaluated under single-task scenarios. By formulating representation learning as an action sequence reconstruction problem and integrating diffusion models, we achieve more stable training and eliminate the need for the time-consuming inference processes typical of these models. This multi-modal approach not only enhances our method's capabilities but also sets our work apart in the field of robotic task learning.
In summary, our approach stands out due to its unique pre-training approach, effective use of multi-view data, and innovative application of diffusion models for multi-modal representation, all contributing to its enhanced generalization and performance in diverse robotic tasks.

Reviewer comment (Experiments):
Given the close similarity to GNFactor, it would be useful to have experiments to analyze which component of NeRFuser is providing the improved performance. The ablation study which changes out stable diffusion for other pretrained models is interesting, but doesn't show the merits of NeRFuser over GNFactor.

Author response:
In our ablation study, we highlight the effectiveness of pre-training 3D representations through neural rendering. This is evident in Table 4, which shows a significant performance gap between NeRFuser and the Learning from Scratch (LfS) approach. This is the key difference between the proposed NeRFuser and the joint learning paradigm of GNFactor.  Our method's success rate substantially decreases without pre-trained 3D representations. We further provide the results of pre-training with data from five unseen tasks. It also achieves better results than GNFactor, showcasing the efficacy of our pre-training approach.

Reviewer comment (Experiments):
A potential weakness of the method, albeit minor, is that it requires a specialized setup during training with multiple calibrated cameras.

Author response:
The proposed NeRFuser greatly improves the line of research that utilizes semantic information for robotic manipulation. [4] requires around 50 calibrated scenes for NeRF training after every action inference. GNFactor [1] leverages generalizable NeRF and does not need per-scene optimization as [4]; however, it requires paired multi-view images and action data for both neural rendering and policy learning, which greatly limits the flexibility of learning a more scalable model. 
NeRFuser, on the other hand, disentangles the neural rendering and the policy learning components, which allows for learning from task-agnostic multi-view datasets to construct semantic representation in a 3D space, and **the training datasets for the policy learning are not required to have multiple views**. This improvement significantly reduces the training and inference time since we only need to query the pre-trained 3D encoder once for the downstream tasks. NeRFuser not only learns a more generalizable policy but also is more time-efficient.

Reviewer comment (Reproducibility):
The quantitative numbers for GNFactor from their paper differ significantly (sometimes as much as 20% success rate) than the ones reported in NeRFuser, could you explain why the performance differs so much in your setup?

Author response:
This performance difference comes from different training settings. We use 50 demonstrations and 200K iterations for GNFactor training while they use 20 demonstrations and 100k in the paper. With more demonstration data, GNFactor achieves better results in our setup.

Reviewer comment (Experiments):
Frames are passed to the system as pointclouds, but there is no mention of how the point clouds are obtained.

Author response:
Thank you for pointing this out. In the simulation environment, we follow previous works like GNFactor and PerAct to obtain RGB and depth images and convert them to point clouds. In the real world, we use the point clouds estimated by RealSense cameras. For details regarding extracting point clouds from a RealSense camera, we invite the reviewer to refer to the documentation: https://dev.intelrealsense.com/docs/docs-get-started.
We have also clarified in the updated manuscript that our point cloud is obtained from the RealSense camera for better clarity.

Reviewer comment (Presentation):
It is just a minor thing, but Table 4 is ambiguous. I think it would be good to supplement it a little more so that it's easy to understand.

Author response:
**A2:** Thank you for pointing this out. We have added a detailed introduction to the caption to indicate that 
LfS indicates a learning-from-scratch baseline without pre-training.
Task-agnostic pre-training indicates that the pre-training phase is trained with out-of-distribution data from 5 unseen tasks.

Reviewer comment (Experiments):
I'm quite interested in the results shown in Figure 5. Could you explain the results? I couldn't find a paragraph mentioning Figure 5. Could you show more visualization results in other settings (unseen, out-of-distribution, without distillation of vision foundation models)?

Author response:
**A3:** Thanks for your interest. Figure 5 shows the novel view synthesis results of NeRFuser on both RGB and diffusion features in the real world. By distilling the 2D stable diffusion model into 3D space, we learn a 3D semantic representation and visualize rendered 2D feature maps. It shows NeRFuser parses all object instances and extracts scene semantics well. In our revised manuscript, we have also added the visualization of 3D point features extracted from a pre-trained 3D encoder using out-of-distribution data in Figure 6.

Reviewer comment (Writing):
The author did not clearly articulate the motivation and contributions of the paper.

Author response:
**Q1:** The author did not clearly articulate the motivation and contributions of the paper.

**A1:** Thank you for your suggestion regarding the clarity of the paper. We have improved the clarity of the introduction sections to emphasize and pinpoint the motivation and contributions of the proposed NeRFuser in the revised manuscript. Here is the summary of the motivation and contributions:

**Motivation:** 

* To perform complex manipulation tasks in the unstructured real world, it is essential to have a comprehensive semantic and geometric understanding of the scene. Previous NeRF-based manipulation methods ([1][2]) mainly focus on learning geometric features but have no semantic understanding of complex scenes. 
* In addition, being able to distinguish different modes in demonstration data is important for multi-task imitation learning. Diffusion policy is a great tool for this but still requires substantial parameter tuning to achieve accurate action prediction. It also needs a time-consuming multi-step denoising process, which results in higher inference latency than non-diffusion behavior cloning methods.

**Contributions:** 

* We leverage NeRF as a 3D pre-training approach to learn a 3D representation that can unify both semantics and geometry. From distilling pre-trained 2D foundation models into 3D space, we learn a generalizable 3D semantic representation with commonsense priors from internet-scale datasets. We show that this pre-trained representation achieves strong out-of-distribution generalization ability. Unlike other NeRF-based methods ([1][2][3]), our method eliminates the need for task-specific in-domain datasets consisting of multi-view images paired with action data for 3D pre-training, which can be expensive to collect in the real world.
* We introduce a novel approach utilizing diffusion training to learn a vision and language feature that distinguishes multi-modality from multi-task demonstrations. By utilizing a feature-conditioned noise predictor and reconstructing the action sequences from different tasks via the diffusion process, the model is capable of distinguishing different modalities and thus improving the robustness and the generalizability of the learned representation. The additionally learned policy network bypasses the need for the time-consuming denoising phase and makes multi-task learning more stable.
* NeRFuser significantly surpasses strong baselines GNFactor and PerAct by over 30% in 10 RLBench tasks and 5 real robot tasks, illustrating the strong generalizability of NeRFuser.

Reviewer comment (Writing):
In my opinion, the writing of the paper is confusing. For example, in the first paragraph of introduction, why are the issues raised considered important and in need of resolution? As a researcher in another field, I cannot determine if this is a consensus within the field or the personal opinion of the author.

Author response:
**Q2:** In my opinion, the writing of the paper is confusing. For example, in the first paragraph of introduction, why are the issues raised considered important and in need of resolution? As a researcher in another field, I cannot determine if this is a consensus within the field or the personal opinion of the author.

**A2:**  First, both semantic and geometric understanding are quite necessary for complex real-world environments ([1][2][3]). For example, in a challenging scene (e.g., partial occlusion, various object shapes, and spatial relationship), comprehending the 3D structure of the scene is important for accurate manipulation and avoiding collision. Understanding the semantics and functionality of various objects is also crucial to a robust and generalizable policy in diverse scenes. Second, modeling the multi-modality of demonstration data has been one of the key challenges in behavior cloning literature ([4][5]). Without identifying the inherent multi-modality in the multi-task demonstrations, the learned policy is prone to overfitting to a small number of demonstration data and difficult to generalize to new scenes with novel objects and arrangements presented.

Reviewer comment (Evaluation):
In Table 1, there seems to be a significant discrepancy between the performance of PerAct and the results reported in the original paper. Is this difference due to variations in the calculation of metrics or because the PerAct used for comparison in the experiments did not fully converge?

Author response:
**Q4:**  In Table 1, there seems to be a significant discrepancy between the performance of PerAct and the results reported in the original paper. Is this difference due to variations in the calculation of metrics or because the PerAct used for comparison in the experiments did not fully converge?  
**A4:**  In PerAct's experiments, they employ 10 and 100 demonstrations for 18 tasks. In our experiment, we apply the same training setting for both peract and our approach by using 50 demonstrations for 10 tasks. To ensure a fair and balanced comparison, we have applied equivalent training configurations for both PerAct and NeRFuser in the paper.

Reviewer comment (Evaluation):
In the original paper, the author argues that the extensive inference time of diffusion models is a significant drawback. However, why is this issue resolved by adding a policy network? Additionally, I did not find any results related to speed or inference time in the experimental section.

Author response:
**Q5:** In the original paper, the author argues that the extensive inference time of diffusion models is a significant drawback. However, why is this issue resolved by adding a policy network? Additionally, I did not find any results related to speed or inference time in the experimental section.

**A5:** To predict an action with a diffusion policy, it is required to go through a “denoising” process so that an accurate action signal can be constructed from a pure Gaussian noise. Previous work [4] takes around 100 denoising steps for this process and each step queries the noise predictor network once. The proposed method utilized the diffusion training to learn the multi-modal representation for observations. Notably we directly predict the corresponding action with a policy MLP network instead of the diffusion model. The size of the policy network is less than 10% of that of the noise predictor network and we only need to query it once for every action inference, which makes the proposed method over 10 times faster in action inference and still benefit from the multi-modality obtained from the diffusion training. Since it involves different implementations such as point sampling empirically, we still observe more than 15 times speedup compared to the diffusion policy.

Reference
[1] Driess, Danny, et al. "Reinforcement learning with neural radiance fields." Advances in Neural Information Processing Systems 35 (2022): 16931-16945.

[2] Li, Yunzhu, et al. "3d neural scene representations for visuomotor control." Conference on Robot Learning. PMLR, 2022.

[3] Ze, Yanjie, et al. "Gnfactor: Multi-task real robot learning with generalizable neural feature fields." Conference on Robot Learning. PMLR, 2023

[4] Chi, Cheng, et al. "Diffusion policy: Visuomotor policy learning via action diffusion." arXiv preprint arXiv:2303.04137 (2023).

[5] Florence, Pete, et al. "Implicit behavioral cloning." Conference on Robot Learning. PMLR, 2022.