Paper: 3D Diffuser Actor: Multi-task 3D Robot Manipulation with Iterative Error Feedback (ICLR-2024)

Reviewer comment (Evaluation):
**No deeper analysis about why diffusion models could help**. The ablation results only show two factors matter, but all these factors seem to be not novel and not surprising, thus deeper analysis might be necessary. I have also checked the supplementary files and the presented Figure 7 looks interesting, but why `scaled linear` is worse than `square cosine` when the former one seems to cover the original distribution better?

Author response:
>No deeper analysis about why diffusion models could help. The ablation results only show two factors matter, but all these factors seem to be not novel and not surprising, thus deeper analysis might be necessary.


Motivated by your comment and the comment of Reviewer Ws6y, and in an attempt to validate the argument regarding utility for diffusion, we conducted an additional experiment where we use a close to **identical** architecture to 3D Diffuser Actor for **regressing** the 3D location and 3D rotation of the target keypose, as opposed to the noise and the use of iterative denoising (no denoising iteration index input for  regression). We show the comparison for 3D location prediction on train and val sets ([here](https://docs.google.com/presentation/d/e/2PACX-1vTZkgzAxzw2IlVY86Ft9AmRC2UR_RBVFV8uFBdeyxeg9hdCWN7mstblmIkOQAogGR2LT1k3dsLEXcTo/pub?start=false&loop=false&delayms=3000)) and the comparison for 3D rotation prediction on train and val sets ([here](https://docs.google.com/presentation/d/e/2PACX-1vSkoF10a1x7D12Ll1xCeJ4rPSA02GzTKPsCq0yS8dJWsZYuUnFAGHjCixD_blR1_hyDBdluIqTtpbsF/pub?start=false&loop=false&delayms=3000#slide=id.g29b18ed0d7c_0_0)). 

As you can see, 3D diffuser actor generalizes much better than the regression model, especially for 3D location.  Both models use the same scene encoder, language encoder, and 3D relative transformer decoder architecture, but one predicts the end-effector through iterative denoising, while the other(regression) through a single feedforward pass.
**Our contribution**: We propose the first 3D keypose diffusion model denoised from 3D scene representations for learning 3D manipulation from demonstrations. We achieve superior performance over many prior methods in apples-to-apples comparisons in an established experimental setup on RLBench(+12% compared to state-of-the-art methods).   Our model, 3D Diffuser Actor, marries 3D representations with diffusion policies for robot manipulation, as  we describe  clearly in the abstract (*"a framework that marries diffusion policies and 3D scene representations for robot manipulation"*) and in the intro. In both abstract and intro  we first describe diffusion policies (in abstract: *"Diffusion policies capture the action distribution conditioned on the robot and environment state using conditional diffusion models. They have recently shown to outperform both deterministic and alternative generative policy formulations in learning from demonstrations."*), then 3D policies (in abstract: *"3D robot policies use 3D scene feature representations aggregated from single or multiple 2D image views using sensed depth. They typically generalize better than their 2D counterparts in novel viewpoints."*) and we say we introduce 3D diffusion policies that marry the above two: (in abstract: *"We unify these two lines of work and present a neural policy architecture that uses 3D scene representations to iteratively denoise robot 3D rotations and translations for a given language task description."*) 
**The specific way this marriage is accomplished is important**:  we represent the current noisy end-effector pose estimate as a 3D token in the 3D feature cloud and featurize it  with relative 3D attentions. Since the ICLR submission deadline, **we ran the exact same model with absolute position attention (closer to what 2D diffusion policies do) on the whole PerAct benchmark and obtained an 8% worse performance**. We consider the above 3D diffusion keypose policy model to be our contribution.

**Please, also see the general response in the beginning of the rebuttal.**



>I have also checked the supplementary files and the presented Figure 7 looks interesting, but why scaled linear is worse than square cosine when the former one seems to cover the original distribution better?

The objective of the noising process is to cover the whole space densely, as during inference we start from pure Gaussian noise and gradually denoise, following the DDPM method. As Figure 7 shows, the scaled linear schedule captures the distribution instead of covering the whole space of possible rotations. As a result, when randomly initializing from Gaussian noise, our model gets out of distribution and cannot denoise to a valid solution.

Reviewer comment (Evaluation):
**The inference time and the denoising steps are both not clear.** The proposed method achieves 12% absolute gain, but considering the inference time of diffusion models, this gain might be not obvious in the real world, due to huge latency. Could the authors also report the wall time between different algorithms?

Author response:
>The inference time and the denoising steps are both not clear. The proposed method achieves 12% absolute gain, but considering the inference time of diffusion models, this gain might be not obvious in the real world, due to huge latency. Could the authors also report the wall time between different algorithms?

Right below we report wall time using a Nvidia GeForce GTX 1080 graphic card.  We use 100 diffusion steps during both training and testing. As we can see, our method is much much slower than Act3d. We are currently working on accelerating inference. The inference time can be drastically reduced using 1) fewer denoising steps, we  suspect that 15 steps may suffice for equivalent performamce based on recent literature (Reuss et al, RSS 2023)  2) more efficient attention implementation (e.g., flash attention [Dao et al, NeuRIPS 2022]). 3) faster diffusion process (e.g. Consistency Model [Song et al, ICML 2023]). We have added the inference time in our manuscript.


||3D Diffuser Actor | Act3D |
|:---------------:|:---------------:|:---------------:|
|Inference Time per keypose  |3 secs | 0.12 secs|

Reviewer comment (Experiments):
**Lack of baselines in real robot experiments**.

Author response:
>Lack of baselines in real-robot experiments.

This is hard to do as it requires resetting the real environment to  identical state across multiple models. PerAct, Act3D, RVT, InstructRL and most other works  do not consider baselines in the real world experiments for that reason.

Reviewer comment (Experiments):
**Lack of multi-task manipulation results in real robot experiments**. Both PerAct [1] and GNFactor [2] have shown ability to execute real-world multi-task manipulation, and it could be good to compare the multi-task performance in real robot also, considering this work is closely related to PerAct [1] and GNFactor [2].

Author response:
>Lack of multi-task manipulation results in real robot experiments. Both PerAct [1] and GNFactor [2] have shown ability to execute real-world multi-task manipulation, and it could be good to compare the multi-task performance in real robot also, considering this work is closely related to PerAct [1] and GNFactor [2].

You are absolutely correct, we did not have the time to do that, it is within the capabilities of our model. We leave this for future work.

Reviewer comment (Writing):
**Typo** in Figure 2 (a): Acter -> Actor

Author response:
>Typo in Figure 2 (a): Acter -> Actor
Thank you for catching this, we fixed it.

Reviewer comment (Novelty):
A portion of the technical method and writing appears to be directly borrowed from [1], but this relationship is not transparently acknowledged.

Author response:
**Please, see the general response in the beginning of the rebuttal.**

Reviewer comment (Novelty):
The proposed framework adopts the 3D scene representation (in a simplified form), 3D relational transformer, and training/evaluation setups from [1], which is never explicitly mentioned in the paper.

Author response:
**Please, see the general response in the beginning of the rebuttal.**

Reviewer comment (Experiments):
The main contribution of this work lies in the use of a diffusion policy to capture multimodal action distributions; however, this aspect is not extensively discussed or thoroughly evaluated.

Author response:
Motivated by your comment and the comment of Reviewer sG3K, and in an attempt to validate the argument regarding utility for diffusion, we conducted an additional experiment where we use a close to **identical** architecture to 3D Diffuser Actor for **regressing** the 3D location and 3D rotation of the target keypose, as opposed to the noise and the use of iterative denoising (no denoising iteration index input for  regression). We show the comparison for 3D location prediction on train and val sets ([here](https://docs.google.com/presentation/d/e/2PACX-1vTZkgzAxzw2IlVY86Ft9AmRC2UR_RBVFV8uFBdeyxeg9hdCWN7mstblmIkOQAogGR2LT1k3dsLEXcTo/pub?start=false&loop=false&delayms=3000)) and the comparison for 3D rotation prediction on train and val sets ([here](https://docs.google.com/presentation/d/e/2PACX-1vSkoF10a1x7D12Ll1xCeJ4rPSA02GzTKPsCq0yS8dJWsZYuUnFAGHjCixD_blR1_hyDBdluIqTtpbsF/pub?start=false&loop=false&delayms=3000#slide=id.g29b18ed0d7c_0_0)). 

As you can see, 3D diffuser actor generalizes much better than the regression model, especially for 3D location.  Both models use the same scene encoder, language encoder, and 3D relative transformer decoder architecture, but one predicts the end-effector through iterative denoising, while the other(regression) through a single feedforward pass.

**Our contribution**: We propose the first 3D keypose diffusion model denoised from 3D scene representations for learning 3D manipulation from demonstrations. We achieve superior performance over many prior methods in apples-to-apples comparisons in an established experimental setup on RLBench(+12% compared to state-of-the-art methods).   Our model, 3D Diffuser Actor, marries 3D representations with diffusion policies for robot manipulation, as  we describe  clearly in the abstract (*"a framework that marries diffusion policies and 3D scene representations for robot manipulation"*) and in the intro. In both abstract and intro  we first describe diffusion policies (in abstract: *"Diffusion policies capture the action distribution conditioned on the robot and environment state using conditional diffusion models. They have recently shown to outperform both deterministic and alternative generative policy formulations in learning from demonstrations."*), then 3D policies (in abstract: *"3D robot policies use 3D scene feature representations aggregated from single or multiple 2D image views using sensed depth. They typically generalize better than their 2D counterparts in novel viewpoints."*) and we say we introduce 3D diffusion policies that marry the above two: (in abstract: *"We unify these two lines of work and present a neural policy architecture that uses 3D scene representations to iteratively denoise robot 3D rotations and translations for a given language task description."*) 
**The specific way this marriage is accomplished is important**:  we represent the current noisy end-effector pose estimate as a 3D token in the 3D feature cloud and featurize it  with relative 3D attentions. Since the ICLR submission deadline, **we ran the exact same model with absolute position attention (closer to what 2D diffusion policies do) on the whole PerAct benchmark and obtained an 8% worse performance**. We consider the above 3D diffusion keypose policy model to be our contribution.


**Please, also see the general response in the beginning of the rebuttal.**

Reviewer comment (Writing):
When building the 3D scene feature cloud, the authors claim, `We associate every 2D feature grid location in the 2D feature maps with a depth value, by averaging the depth values of the image pixels that correspond to it.` What is a `grid` here? Is each 2D feature map from the feature pyramid network separated into an NxN grid akin to ViT?

Author response:
An image encoder extracts a $H \times W \times C$ feature map from each camera view, where $H$, $W$, and $C$ denotes the height, width and channel dimension of the feature map.  Each $1 \times 1 \times C$ feature vector (we call it token) in the feature map corresponds to a fixed-size patch in the input image. We can thus lift a 2D feature token to 3D by averaging the 3D locations of all pixels within the patch. The 3D locations of pixels are obtained by combining the 2D pixel locations and depth values (x, y, d) using the camera intrinsics and pinhole camera equation. We updated the manuscript and Figure 1 to explain the feature extraction and 2D-3D lifting procedure more clearly.

Reviewer comment (Reproducibility):
When generating the 3D feature cloud, how does the method handle cases where multiple points (pixels) from different views correspond to the same 3D location? Are the features averaged in such instances?

Author response:
We found that having two points with identical 3D locations rarely happens, so we did not explicitly handle this case. Although some merging could be applied, such as averaging, we adopted a more general solution and simply let attention focus on the important features.

Reviewer comment (Writing):
What constitutes a visual token in this context? Are individual 3D points considered tokens?

Author response:
Each  feature vector from the 2D feature maps lifted to 3D is a 3D token.

Reviewer comment (Experiments):
Given that diffusion policies suggest the utility of diffusing multiple action steps into the future, has the method been evaluated with multiple keypoint steps in the diffusion process?

Author response:
Thank you for your suggestion. Although in this version we only predict the next keypose, our model could indeed support the prediction of multiple keyposes and it would be interesting to explore that in the future as a way to plan. Nonetheless, we would like to emphasize that our diffusion model predicts the next subgoal to reach, not low-level actions directly. Instead, once a subgoal keypose has been predicted, then a low-level planner is assigned with predicting a sequence of actions that connect the current pose and the predicted keypose. This decomposition has been followed by previous approaches as well (e.g. C2F-QA, PerAct, Act3D). This is different from what related works on diffusion policies do, which is to predict low-level actions directly.

Reviewer comment (Novelty):
The scientific contribution of this paper is unclear. Diffusion model could not be the contribution of this paper, and adopting diffusion mode for trajectory generation is not a new idea (see [a]).

Author response:
**Our contribution**: We propose the first 3D keypose diffusion model denoised from 3D scene representations for learning 3D manipulation from demonstrations. We achieve superior performance over many prior methods in apples-to-apples comparisons in an established experimental setup on RLBench(+12% compared to state-of-the-art methods).   Our model, 3D Diffuser Actor, marries 3D representations with diffusion policies for robot manipulation, as  we describe  clearly in the abstract (*"a framework that marries diffusion policies and 3D scene representations for robot manipulation"*) and in the intro. In both abstract and intro  we first describe diffusion policies (in abstract: *"Diffusion policies capture the action distribution conditioned on the robot and environment state using conditional diffusion models. They have recently shown to outperform both deterministic and alternative generative policy formulations in learning from demonstrations."*), then 3D policies (in abstract: *"3D robot policies use 3D scene feature representations aggregated from single or multiple 2D image views using sensed depth. They typically generalize better than their 2D counterparts in novel viewpoints."*) and we say we introduce 3D diffusion policies that marry the above two: (in abstract: *"We unify these two lines of work and present a neural policy architecture that uses 3D scene representations to iteratively denoise robot 3D rotations and translations for a given language task description."*)  
**The specific way this marriage is accomplished is important**:  we represent the current noisy end-effector pose estimate as a 3D token in the 3D feature cloud and featurize it  with relative 3D attentions. Since the ICLR submission deadline, **we ran the exact same model with absolute position attention (closer to what 2D diffusion policies do) on the whole PerAct benchmark and obtained an 8% worse performance**. We consider the above 3D diffusion keypose policy model to be our contribution.  
**Please, also see the general response in the beginning of the rebuttal.**

Reviewer comment (Presentation):
The figures in this paper lack sufficient illustrative value and information. For instance, Figure 1 appears to show the model taking a multi-view image as input, yet the caption indicates it uses a 3D scene feature cloud. A clear connection between these two elements would greatly improve understanding.

Author response:
An image encoder extracts a $H \times W \times C$ feature map from each camera view, where $H$, $W$, and $C$ denotes the height, width and channel dimension of the feature map.  Each $1 \times 1 \times C$ feature token in the feature map corresponds to a fixed-size patch in the input image. We can thus lift a 2D feature token to 3D by averaging the 3D locations of all pixels within the patch. The 3D locations of pixels are obtained by combining the 2D pixel locations and depth values (x, y, d) using the camera intrinsics and pinhole camera equation. We updated the manuscript to explain the feature extraction and 2D-3D lifting procedure more clearly. We also **updated Figure 1** to illustrate the lifting process.

Reviewer comment (Reproducibility):
Your diffusion model appears to produce only the target pose of the action, after which a trajectory is generated using the MoveIt planner given the initial joint state and target joint state. Does the robot execute the entire trajectory directly until the end, or does it update the target pose using the diffusion model and regenerate the trajectory after each forward step?

Author response:
Once a keypose is predicted, a low-level planner is tasked to predict a trajectory between the current and the predicted pose. This trajectory is executed open-loop until the end. We found that this was sufficient for all 18 tasks we focus on. At the same time, it’s more efficient computationally, since the denoising process is sparsely called.