Paper: Generating Robot Policy Code for High-Precision and Contact-Rich Manipulation Tasks (ICLR-2024)

Reviewer comment (Evaluation):
Even though the paper emphasizes on continuous aspect of the prediction, there is not much continuous quantities going on in the predicted codes. The values of floats in the generated code does not have much variation. The paper gives the impression that it can predict full range of continuous values but in the examples the continuous values can be looked at as a value drawn from a very small set such as `{0, 0.01, 0.02, 0.04, 0.001}`.

Author response:
We realized that this was indeed unclear in the paper and we'll fix it in the following way: the goal of our initial experiment was to probe the capabilities of LLMs for high-precision prediction and address a shortcoming present in prior work [1]. We are working on demonstrating the importance of these probing experiments in performing high-precision tasks in real-world robotics scenarios during the rebuttal period.

Reviewer comment (Experiments):
The part that really needs continuous value is abstracted away as `pose(1)` and the model just apply relative transforms to that. If `pose(1)` is chosen close to the final poses, the LLM does not need to do much to accomplish the task. It would help if the paper reports the distribution of `pose(1)` distance to the final target poses.

Author response:
The choice of pose is kept constant across all baselines. In spite of this, the task is difficult to achieve without the correct action space because successful routing and unrouting requires reasoning over interaction forces. This is why the point-to-point baseline is only 20% successful on average across cable routing tasks. Even with a fixed pose, there is randomization in the orientation of the cable at the point of grasping.

Reviewer comment (Experiments):
Not all the prompt categories are used in the example prompt. I am particularly interested to see a prompt that uses Spatial Patterns. I also believe that this is a simplistic prompt. The resolution of patterns can make a big difference in success or failure of tasks. In addition, how are the continuous orientation of parts represented by characters?

Author response:
That is correct. We are working on adding examples using the spatial patterns prompt during the rebuttal period. The main purpose of this prompt is to relay high level spatial information about the relative position of objects in the scene. It doesn’t contain information about the continuous orientation of parts represented by the characters.

Reviewer comment (Theory):
The paper assumes that perception is a solved problem and it does not deal with uncertainties in the perception.

Author response:
We agree that incorporating perception is an important step for future work and have added this into the conclusion.

Reviewer comment (Reproducibility):
What is total number of trials for Table 1 and 2?

Author response:
Following the protocol of VoxPoser [2] in Table 1, we report the average success over 10 evaluation runs. We report results on the best of 5 generated code examples. We’ve highlighted this experimental detail in the revised PDF: see the first line of section 5.2.3.

Reviewer comment (Evaluation):
Claims: the 3x and 4x improvement was a bit oversold. I recommend authors to focus on the improvement over the best alternative approach rather than the point to point.

Author response:
We’ve modified that abstract to make this distinction more clear. To clarify, the point-to-point method is most analogous to prior work such as Code-as-Policies. The best alternative approach (Fixed Compliance) is an ablation of our proposed action space that does not include termination constraints or modifiable compliance parameters. Generating robot policy code with Fixed Compliance has not been tried in prior work; we are the first to do so. In the updated PDF, we have changed the names of the baselines and ablations to make this more clear.

Reviewer comment (Novelty):
Novelty: The paper introduces limited novelty compared to the previous work. The main contribution is to allow LLMs to also generate constraints for their manipulation functions.

Author response:
It has been long discussed in the robotics community whether LLM advancement could ever result in contact-rich dexterous control and this is the first paper showing signs that this is indeed possible. We develop a system for automatically generating robot policy code for dexterous tasks by allowing LLMs to specify constraints on the stiffness, forces, and trajectories required to perform contact-rich manipulation tasks and show that our approach outperforms the state-of-the-art approaches such as Code-as-Policies by a significant margin, which uses a contact-unaware model.

Reviewer comment (Experiments):
Generalizability: given the few-shot learning example in the appendix, the prompt holds key sections for solving the task (e.g. move up unless snag is detected). It would have been great if authors included only examples not exactly needed to solve the task for the second domain. We see with no examples, LLM did not do great specially on the harder task of routing the cable.

Author response:
We agree that adding a targeted generalization study could help improve the overall quality of the paper and are working to add in more experiments that demonstrate this by the end of the rebuttal period. Our finding that the zero-shot prompt performs poorly is consistent with prior work on LLMs and robotics. Most work in this area uses a large number of prompt examples. For example VoxPoser [1] leverages 8 different prompts with an average of ~9 subcommands per prompt. See an example here: https://github.com/huangwl18/VoxPoser/blob/main/src/prompts/rlbench/composer_prompt.txt. For code generation benchmarks, Code-As-Policies [2] uses a prompt with 24 pre-defined functions: https://github.com/google-research/google-research/blob/master/code_as_policies/Experiment_%20Robot%20Code-Gen%20Benchmark.ipynb. Our main goal in this experiment is to test the composition of primitives in solving a cable routing task, which is why we provide 3 sub-commands in the same order for each task. The LLM successfully generates compositions of the subcommands to solve the task. For the cable-unrouting task, the generated code also includes newly defined constraints that detect rightward force. This example is present on the paper website and we’ve updated the appendix to include this as well.

Reviewer comment (Experiments):
Figure 6 a,b: For each point generated by LLM, is the input true history, or is it based on previous points generated by LLM. My understanding if the former, but would be great to clear this out in the paper.

Author response:
In Figure 6b, all of the visualized orange points are generated from previous points generated by the LLM. In Figure 6a, the x-axis represents the number of true pairs input and the y-axis represents the average prediction error of the next predicted y value. We have updated the draft to make this clear.

Reviewer comment (Writing):
"every command is an unseen command.", I think you meant unseen in the sense of not seeing examples of it but the command is defined as an input through the prompt and hence seen.

Author response:
Yes. To be specific, the command is the last section of the prompt. We do not provide paired code examples for the given command.

Reviewer comment (Reproducibility):
"We also tune the following properties independently for each model". How did you optimize them?

Author response:
We’ve updated Section 5.2.3 to be more precise. The insertion depths are manually tuned by a human operator once for a given environment and then specified in the prompt. This is to make the Point-to-Point baseline as strong as possible when comparing against our method. Otherwise, the Point-to-Point policy will cause the robot to fault at greater insertion depths because it will continue to apply force once in contact.

Reviewer comment (Evaluation):
Figure 8: While the type of errors made by LLM changed as more hints were introduced the total number of errors seemed to remain constant. This means the hint type did not change the success rate of the agent. Is that true? If so, please make it explicit it in the paper.

Author response:
In Figure 8, we perform rejection sampling to estimate the likelihood of each error type given that the generated code fails. The hint type does change the success rate of the agent. We have updated the paper to be more clear.

Reviewer comment (Reproducibility):
Can you provide details on the number of shots used for few-shot experiments?

Author response:
The prompt contains 3 examples where the control API is called with relevant subcommands: moving down until contact is reached, moving up unless a snag is detected, and moving right unless a snag is detected. We count each example subcommand as a single shot. Please see the updated Section 5.2.2 where this is clarified.

Reviewer comment (Experiments):
The paper does not demonstrate LLM policies are better than other policies, because it does not compare against those baselines. In this context, the paper lacks discussion about _why_ LLM policies might be preferable.

Author response:
Compared to policy learning approaches that rely on reinforcement learning or imitation learning, our approach is far more data efficient and requires less time from human operators. We explain these differences concretely below using some of the references you provided (thank you for bringing them to our attention). Please also see the updated related work where we add these comparisons.

- **Imitation learning:** in the contact-rich push-T task, Diffusion Policies [1] show generalization across the initial position of the T block across 136 demonstrations. Before the data collection can begin  “the operator has [already] performed this task for many hours” to reach the proficiency needed for demonstrations of a sufficient quality. 
- **Reinforcement learning:** in "Zero-Shot Transfer of Haptics-Based Object Insertion Policies" the pre-training phase requires building a dedicated simulator and ~9M simulated environment steps (from Table III of their paper: 64K random steps to seed training + [64K training iterations x 140 steps per training iteration]). "Deep reinforcement learning for industrial insertion tasks with visual inputs and natural rewards" requires 2.5k-10k real world timesteps per task. "Factory: Fast contact for robotic assembly" also requires a dedicated simulator and >2k environment steps to reach success in simulation and is not validated in the real world.

By contrast, our experiments only require tuning the insertion depth specified within the prompt and a natural language command to request a new task. To draw a fair comparison to reinforcement learning approaches, we’d need to restrict the setting to ~3 demonstrations, which would result in 0% success rate. Our results on the Functional Manipulation Benchmark show that one generalization benefit to using a LLM is that we can leverage the language model’s world knowledge about different object types to generalize in a zero-shot fashion. We are working on adding more NIST-style insertion experiments that you described to further demonstrate task generalization.

[1] Chi et. al. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. RSS 2023.

Reviewer comment (Evaluation):
If some other reasons make LLM policies more attractive, please discuss those reasons.

Author response:
Compared to policy learning approaches that rely on reinforcement learning or imitation learning, our approach is far more data efficient and requires less time from human operators. We explain these differences concretely below using some of the references you provided (thank you for bringing them to our attention). Please also see the updated related work where we add these comparisons.

- **Imitation learning:** in the contact-rich push-T task, Diffusion Policies [1] show generalization across the initial position of the T block across 136 demonstrations. Before the data collection can begin  “the operator has [already] performed this task for many hours” to reach the proficiency needed for demonstrations of a sufficient quality. 
- **Reinforcement learning:** in "Zero-Shot Transfer of Haptics-Based Object Insertion Policies" the pre-training phase requires building a dedicated simulator and ~9M simulated environment steps (from Table III of their paper: 64K random steps to seed training + [64K training iterations x 140 steps per training iteration]). "Deep reinforcement learning for industrial insertion tasks with visual inputs and natural rewards" requires 2.5k-10k real world timesteps per task. "Factory: Fast contact for robotic assembly" also requires a dedicated simulator and >2k environment steps to reach success in simulation and is not validated in the real world.

By contrast, our experiments only require tuning the insertion depth specified within the prompt and a natural language command to request a new task. To draw a fair comparison to reinforcement learning approaches, we’d need to restrict the setting to ~3 demonstrations, which would result in 0% success rate. Our results on the Functional Manipulation Benchmark show that one generalization benefit to using a LLM is that we can leverage the language model’s world knowledge about different object types to generalize in a zero-shot fashion. We are working on adding more NIST-style insertion experiments that you described to further demonstrate task generalization.

[1] Chi et. al. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. RSS 2023.

Reviewer comment (Experiments):
The impact of hint inputs (Section 4.1(3)) is not experimentally validated. This is important, because from the text it seems like the hints provide strong policy guidance like "do a grid pattern search".

Author response:
We will work on adding an ablation of hint inputs. We will try to add these by the end of the rebuttal period.

Reviewer comment (Experiments):
Impact of noise in state description provided to the LLM (e.g., imperfectly sensed spatial pattern of the board in Fig. 3(d), or noisy robot starting pose) is not examined experimentally.

Author response:
We agree that a precise analysis of the impact of noise is important and can work to study this in the future. Currently, there are two sources of noise in our experiments. In the insertion experiments, the star-shaped peg’s initial position is sampled uniformly from 0 to pi/2 and the half-pipe shaped peg is sampled uniformly from 0 or pi. In the cable manipulation, there is noise in the orientation of the cable within the grasp of the gripper and the tautness of the cable varies across episodes. This occurs from minor displacements of the untethered cable at the start of an evaluation episode.

Reviewer comment (Experiments):
Is the scripted policy for the insertion task unfairly weak compared to the LLM? It is tuned to the easiest object `circle` and expected to generalize to `star` and `half-pipe`. On the other hand, the LLM policy is provided with a hint about which object it is planning the policy for.

Author response:
It is correct that we tune the insertion parameters on the circle example. Our goal with this experiment is to show that expert policies require task-specific tuning and don’t work out-of-the-box when the task parameters change. By contrast, LLM-generated policy code only requires a natural language task description.

Reviewer comment (Reproducibility):
Please provide the policies generated by the expert scripter and the LLM for the insertion task. This will allow readers to compare the LLM output to the human-designed scripted policy and also allow the reader to know which hints were required by the LLM.

Author response:
Please see the updated website (https://dex-code-gen.github.io/dex-code-gen/).

Reviewer comment (Evaluation):
Please clarify the evaluation protocol, because the language is unclear, especially the last sentence of Section 5.2.3.

Author response:
Thank you for pointing this out. Because the Point-to-Point model has no compliance, it will continue pushing down to make contact until a fault is reached. For example, our prompt contains example pose translations to make contact with the surface, such as `Pose3(translation=(0, 0, -0.02))` where 0.02 is the insertion depth. The compliant policy will be robust to different insertion depths, but the Point-to-Point policy will cause the robot to fault at greater insertion depths because it will continue to apply downward force once in contact. We tune the insertion depth to make the Point-to-Point baseline as strong as possible when comparing against our method. We have updated Section 5.2.3 to be more precise. Please see the updated PDF for this change.

Reviewer comment (Evaluation):
Please explain how the potential generalization advantages of LLM policies are shown in the current experiments.

Author response:
In the FMB experiments in table 1, we see that our approach is able to generalize across different object geometries with only high level object descriptions provided to the LLM. By contrast, an expert scripted policy does not work out of the box across geometries. In the RGMCS results, we find that the LLM is able to generate a performant policy for two different cable manipulation tasks. The only change to the prompt across each scenario is the final user command specified.

Reviewer comment (Theory):
Most of contact-rich manipulation tasks end up to be partially observable due to contact formations and unobservability of the contact states. This aspect of the problem can not be handled in the proposed method. This makes the proposed method a rather ad-hoc attempt at solving the problem than a principled way of incorporating LLM for decision making during contact-rich tasks.

Author response:
_Most of contact-rich manipulation tasks end up to be partially observable due to contact formations and unobservability of the contact states. This aspect of the problem can not be handled in the proposed method. This makes the proposed method a rather ad-hoc attempt at solving the problem than a principled way of incorporating LLM for decision making during contact-rich tasks._

We agree that incorporating more perception is an important step for future work. We’ve highlighted this discussion in the conclusion of the updated PDF. We find that for these tasks our approach is capable of generating performant policies from force readings alone.

Reviewer comment (Presentation):
From a presentation point of view, the authors need to clarify what their decision variables are which are exposed to the LLM. Having that specified will help with understanding about the reasoning performed by LLMs.

Author response:
_From a presentation point of view, the authors need to clarify what their decision variables are which are exposed to the LLM. Having that specified will help with understanding about the reasoning performed by LLMs. Is the impedance the only parameter exposed to the LLM?_

We’ve updated our website  (https://dex-code-gen.github.io/dex-code-gen/) with exact prompts and generated code examples to clarify which action spaces are made available to the LLM.

Reviewer comment (Presentation):
From a presentation point of view, the authors need to clarify what their decision variables are which are exposed to the LLM. Having that specified will help with understanding about the reasoning performed by LLMs.

Author response:
_From a presentation point of view, the authors need to clarify what their decision variables are which are exposed to the LLM. Having that specified will help with understanding about the reasoning performed by LLMs. Is the impedance the only parameter exposed to the LLM?_

We’ve updated our website  (https://dex-code-gen.github.io/dex-code-gen/) with exact prompts and generated code examples to clarify which action spaces are made available to the LLM.

Reviewer comment (Reproducibility):
Is the impedance the only parameter exposed to the LLM?

Author response:
_From a presentation point of view, the authors need to clarify what their decision variables are which are exposed to the LLM. Having that specified will help with understanding about the reasoning performed by LLMs. Is the impedance the only parameter exposed to the LLM?_

We’ve updated our website  (https://dex-code-gen.github.io/dex-code-gen/) with exact prompts and generated code examples to clarify which action spaces are made available to the LLM.

Reviewer comment (Reproducibility):
How is the LLM tuning the impedance parameter? Are you allowing LLM to interact with the current system? or its just done using a rule that LLM comes up with?

Author response:
_How is the LLM tuning the impedance parameter? Are you allowing LLM to interact with the current system? or its just done using a rule that LLM comes up with?_

Please find an example of the admittance move doc-string below. This doc-string is passed to the LLM as part of the prompt. 
```
"""
- cartesian_admittance_move: This moves the robot to a target_pose until a termination condition is reached.
  Args:
      max_cartesian_stiffness:
          The maximum allowed stiffness along each cartesian dof (6d), expressed in
          the robot base frame.
      target_impedance:
          (0,1] 6d-vector specifying the target impedance along each cartesian dof.
      target_pose:
          Target pose for the robot flange frame in the base frame.
      termination_condition:
          Termination condition.
      virtual_cartesian_inertia:
          The diagonal representation of the desired virtual Cartesian inertia
          matrix, expressed in the robot base frame [kg, kg m^2]
      execution_timeout_seconds:
          Timeout for execution. Defaults to 30s if not specified
          Default value: 10.0
      tare_ft_sensor: False when in contact, True otherwise.
"""
```
The LLM outputs policy code that parametrizes this method. For example:
```
cartesian_admittance_move(
        max_cartesian_stiffness=[1000, 1000, 1000, 300, 300, 300],
        target_impedance=[0.1, 0.1, 0.1, 0.1, 0.1, 0.1],
        target_pose=target_pose,
        termination_condition=search_termination_condition,
        virtual_cartesian_inertia=[1, 1, 1, 1, 1, 1],
        execution_timeout_seconds=10.0,
        tare_ft_sensor=False
    )
```
We’ve updated the paper website to include more examples of this process.