Paper: Creative Robot Tool Use with Large Language Models (ICLR-2024)

Reviewer comment (Experiments):
Although this paper focus on high-level planning using tools, the provided descriptions are too detailed on the targetted tools, which makes it hard to see if the hints make LLM propose the solutions. For example, for the Milk-Reaching example, the hammer is provided with detailed instructions on how to grasp, the descriptions on its layout, while other objects are not described that detailedly. Such bias can make the results unfair. And in all 6 experiments, the number of objects in the descriptions are limited, it may be not hard for the LLM to pick a related object.

Author response:
> Q1.2: In milk-reaching experiment, the number of objects in the descriptions is limited. It may not be hard for the LLM to pick a related object.
  - We increase the number of objects that may serve as tools to five, including a hammer, a pineapple toy, a lock, a tomato toy, and a cube. We also provide the same amount of information for each object, including (1) graspable points for non-rectangular objects and the position of rectangular objects and (2) bounding box sizes of each object. Considering the non-rectangular shape of the hammer and pineapple toy, we added a description related to the orientation. However, the orientation description does not affect the tool selection in experiments. After rerunning experiments, we noticed that RoboTool can still correctly select the hammer as the tool among ten runs.  
 - Besides selecting the correct tool, how and when to use the tool are also important. For example, the Can-Grasping task has three objects, five skills (2 have a discrete action parameter, 1 has continuous action parameters), and 15 planning steps. The underlying complexity of planning is significant, and any errors in the intermediate steps could lead to a catastrophic failure. Despite the difficulties, RoboTool achieved a high success rate across different domains, indicating it is not just randomly picking an object to interact with.

Reviewer comment (Experiments):
The descriptions are sometimes confusing. For example, in the Cube-Lifting, the cube weight is 10kg, and the robot weight is also 10kg, then in the video, why the robot will fall down so quickly when it goes to another side. It’s a bit confusing if the description reflects the real property and why not use the real physical attributes.

Author response:
> Q3: You should give the same set of attributes to all objects in the tasks:
 - Thanks for the suggestion. We have cleaned up all the task descriptions. For robot arm experiments, the task description contains (1) graspable points for non-rectangular objects and the position of rectangular objects and (2) bounding box sizes of each object. For quadrupedal robot experiments, the task description contains each object's center position and size. Since tasks in quadrupedal robot experiments are loco-manipulation tasks, we also include the weight of each object in the scene.
     - **Imbalance description in Milk-Reaching; Inaccurate cube weight of Cube-Lifting.** Thanks for the insightful question. We have cleaned up the object description and changed the cube weight to the actual value. 
 - After rerunning the experiments, we noticed that RoboTool has a slight performance drop (0.87 -> 0.83), but it is insignificant.

Reviewer comment (Experiments):
For some constraints mentioned in the description, it’s unclear why the constraints make sense. For example, in the Cube-Lifting example, in the constraints, “you can push the chair only in the x-direction”, it makes readers confusing if the input description is well-tuned for the specific example and how such things make the system generalizable across different tasks.

Author response:
> Q2: Description for constraints, e.g., 'you can push the chair only in the x-direction'. Are they important? Could they be removed? 

 - Without this constraint, RoboTool still produces the same high-level plan skeleton, but with a difference in the target position to push the chair. If the chair is pushed in the y-direction, the robot may clash with the setup, and the surfboard will fall on top of the robot and fail the task. This is related to the realistic setup and the embodiment of the robot rather than our general framework.
 - Correct constraints are critical to success in such challenging tasks. Traditional TAMP literature encodes kinematics and dynamics constraints as predicates and logical rules. For example, in [6], in order to impose constraints on the action primitive ```grasp(X, Y)```, ```[inside X Y] (staFree X Y)``` refers to be predefined, where ```[inside X Y]``` defines point ```X``` is inside object ```Y```, and ```(staFree X Y)``` refers stable (constrained to zero velocity) free joints from ```X``` to ```Y```. In cognitive robotics, there is a rich body of work that requires identifying, modeling, and learning constraints, for example, learning constraints from human feedback [7]. Inputing constraints is not unique to our method but rather common in the robotics literature. Similar to [8, 9], the user provides free-form natural language rather than demonstrations or predefined predicates and logical rules.
 - With that being said, beyond explicit language descriptions, we think there are indeed more seamless ways to obtain such knowledge through interactions and encode them, such as physically grounded Vision-Language Models [10]. However, there are currently no VLMs that can directly provide grounded constraints information. Nevertheless, RoboTool is one of the first steps towards exploring the environment- and embodiment-related constraints and incorporating them into decision-making.

Reviewer comment (Experiments):
Regarding the benchmark, are there additional results showcasing varying object layouts for each demonstration while maintaining a consistent description format?

Author response:
> Q1.1: Are the object layouts changed?

Thanks for the question. The positions are indeed randomized. For robot arm experiments, we randomize the initial configuration of objects. We randomize the initial position of movable objects and the robots for quadrupedal robot experiments. We show the randomization ranges in Table 1 and add it to Appendix G in the updated draft.

Table 1: Randomization of objects in each task.
| | Objects: Randomization range [x_min, x_max, y_min, y_max] |
| ------------- | ------- |
| Milk-Reaching | pineapple toy, lock, cube, tomato toy: [0, 0.45, -0.25, 0.25]; hammer: [0, 0.45, -0.25, 0]; milk: [0.45, 0.6, 0, 0.25]  |
| Can-Grasping | stick: [0, 0.45, -0.25, 0]; paper: [0, 0.45, 0, 0.25]; can: [0.45, 0.7, -0.1, 0.1]  |
| Button-Pressing | magnetic_cube1, magnetic_cube2, magnetic_cube3: [0, 0.1, -0.25, 0.25]; button: [0.45, 0.7, -0.25, 0.25]  |
| Sofa-Traversing | surfboard: [0.3, 0.6, -0.175, 0.0]; cloth: [0.3, 0.6, 0.0, 0.2]; robot: [-0.5, -0.2, -0.2, 0.2];   |
| Sofa-Climbing | box_2: [0.0, 2.0, 0.0, 1.0]; robot: [0.0, 3.0, 0.0, 2.0];   |
| Cube-Lifting | chair: [0.8, 1.2, -0.1, 0.1]; surfboard: [0.8, 1.2, 0.3, 0.5]; robot: [0.0, 3.0, 0.0, 2.0];   |

Reviewer comment (Experiments):
In the demos, how does the system respond when constraints in the input description are removed? How sensitive is it to changes in the description?

Author response:
> Q2: Description for constraints, e.g., 'you can push the chair only in the x-direction'. Are they important? Could they be removed? 

 - Without this constraint, RoboTool still produces the same high-level plan skeleton, but with a difference in the target position to push the chair. If the chair is pushed in the y-direction, the robot may clash with the setup, and the surfboard will fall on top of the robot and fail the task. This is related to the realistic setup and the embodiment of the robot rather than our general framework.
 - Correct constraints are critical to success in such challenging tasks. Traditional TAMP literature encodes kinematics and dynamics constraints as predicates and logical rules. For example, in [6], in order to impose constraints on the action primitive ```grasp(X, Y)```, ```[inside X Y] (staFree X Y)``` refers to be predefined, where ```[inside X Y]``` defines point ```X``` is inside object ```Y```, and ```(staFree X Y)``` refers stable (constrained to zero velocity) free joints from ```X``` to ```Y```. In cognitive robotics, there is a rich body of work that requires identifying, modeling, and learning constraints, for example, learning constraints from human feedback [7]. Inputing constraints is not unique to our method but rather common in the robotics literature. Similar to [8, 9], the user provides free-form natural language rather than demonstrations or predefined predicates and logical rules.
 - With that being said, beyond explicit language descriptions, we think there are indeed more seamless ways to obtain such knowledge through interactions and encode them, such as physically grounded Vision-Language Models [10]. However, there are currently no VLMs that can directly provide grounded constraints information. Nevertheless, RoboTool is one of the first steps towards exploring the environment- and embodiment-related constraints and incorporating them into decision-making.

Reviewer comment (Experiments):
Why not for all objects, give the same set of attributes no matter if the attributes are useful enough? In this way, it can better show if the system is able to really extract the useful information from the descriptions without hints.

Author response:
> Q3: You should give the same set of attributes to all objects in the tasks:
 - Thanks for the suggestion. We have cleaned up all the task descriptions. For robot arm experiments, the task description contains (1) graspable points for non-rectangular objects and the position of rectangular objects and (2) bounding box sizes of each object. For quadrupedal robot experiments, the task description contains each object's center position and size. Since tasks in quadrupedal robot experiments are loco-manipulation tasks, we also include the weight of each object in the scene.
     - **Imbalance description in Milk-Reaching; Inaccurate cube weight of Cube-Lifting.** Thanks for the insightful question. We have cleaned up the object description and changed the cube weight to the actual value. 
 - After rerunning the experiments, we noticed that RoboTool has a slight performance drop (0.87 -> 0.83), but it is insignificant.

Reviewer comment (Experiments):
Although this work try demonstrating the tool usage ability in the high-level setting, it’s hard to ignore the influence from different details. How to make sure the description describe the scene without heavy human designing?

Author response:
> Q4 & Q5: How to make sure the description describes the scene without heavy human design? Does the description hint at the final plan?
 - We use human-designed templates due to the limitation of existing Vision Language Models. As mentioned in the response to Q1.3, the SOTA VLM models are insufficient in 3D reasoning, and it is hard to get a reasonable estimate of object positions and sizes directly without the help of existing APIs. Moreover, our tasks require information about the constraints and weights, which are hard to get from VLMs. Hence, to explore the complex creative tool-use tasks with existing foundation models, we have no choice but to provide a template description.
 - Based on the reviewer's suggestion in Q4, we have cleaned up the description and made sure that different objects in the scene are provided with similar amounts of information, such as positions, sizes, and weights. Please note that the provided information is sufficient but may be unnecessary for finishing the tasks. In other words, we have provided distractors, such as other objects in tool-selection tasks, which require LLMs to filter the information and reason based on the most important information. 
 - Even with the same amount of information, the state-of-the-art baselines still have low success rates, as shown in Table 2. RoboTool performs better than baselines by a large margin, highlighting the task complexity of creative tool-use problems.

Table 2: Success rates of RoboTool and additional baselines.
|  Task | RoboTool (ours) | CaPs | ViperGPT |
| --- | -------- | -------------- | -------- |
|  Milk-Reaching  | 0.9 | 0.0 | 0.0 |
|  Can-Grasping  | 0.7 | 0.2 | 0.0 |
|  Button-Pressing  | 0.8 | 0.6 | 0.6 |
|  Sofa-Traversing  | 1.0 | 0.4 | 0.3 |
|  Sofa-Climbing  | 1.0 | 0.4 | 0.2 |
|  Cube-LIfting  | 0.6 | 0.4 | 0.0 |
|  Average  | 0.83 | 0.33 | 0.18 |

Reviewer comment (Experiments):
How to make sure the given grasping point and other attributes or additional description for one object are not hints to the tool usage demo?

Author response:
> Q4 & Q5: How to make sure the description describes the scene without heavy human design? Does the description hint at the final plan?
 - We use human-designed templates due to the limitation of existing Vision Language Models. As mentioned in the response to Q1.3, the SOTA VLM models are insufficient in 3D reasoning, and it is hard to get a reasonable estimate of object positions and sizes directly without the help of existing APIs. Moreover, our tasks require information about the constraints and weights, which are hard to get from VLMs. Hence, to explore the complex creative tool-use tasks with existing foundation models, we have no choice but to provide a template description.
 - Based on the reviewer's suggestion in Q4, we have cleaned up the description and made sure that different objects in the scene are provided with similar amounts of information, such as positions, sizes, and weights. Please note that the provided information is sufficient but may be unnecessary for finishing the tasks. In other words, we have provided distractors, such as other objects in tool-selection tasks, which require LLMs to filter the information and reason based on the most important information. 
 - Even with the same amount of information, the state-of-the-art baselines still have low success rates, as shown in Table 2. RoboTool performs better than baselines by a large margin, highlighting the task complexity of creative tool-use problems.

Table 2: Success rates of RoboTool and additional baselines.
|  Task | RoboTool (ours) | CaPs | ViperGPT |
| --- | -------- | -------------- | -------- |
|  Milk-Reaching  | 0.9 | 0.0 | 0.0 |
|  Can-Grasping  | 0.7 | 0.2 | 0.0 |
|  Button-Pressing  | 0.8 | 0.6 | 0.6 |
|  Sofa-Traversing  | 1.0 | 0.4 | 0.3 |
|  Sofa-Climbing  | 1.0 | 0.4 | 0.2 |
|  Cube-LIfting  | 0.6 | 0.4 | 0.0 |
|  Average  | 0.83 | 0.33 | 0.18 |

Reviewer comment (Novelty):
Lack of novelty: works such as [1], [2], [3], [4] and [5] have taken similar approaches to neuro-symbolic learning and robotic manipulation, via LLM-generated programs or TAMP structures. Moreover, it's not particularly satisfying to me that the entire method interacts only with GPT-4 at the API level. The paper in effect becomes a "prompt engineering" work, which, while interesting, does not meet the bar for original technical contribution at ICLR.

Author response:
> Novelty of the paper.

We respectfully disagree with the reviewer about the novelty of RoboTool. Here is a summary of our contributions:
 - **Novel Problem:** Creative tool use is an important yet under-explored problem in robotics. It will greatly expand the capability of robots to solve tasks that are impossible originally. 
 - **Novel Benchmark:** We designed a novel challenging benchmark to test specifically the creative tool use behavior, which contains two robot embodiments and three categories of creative tool use. Such a benchmark is highly needed and non-trivial to design [1].
 - **Novel Method:** Although the planner-coder LLM agent has been explored in existing literature, blindly using existing API-based methods such as Code as Policy fails to address the challenges posed by creative tool use in terms of cognition, reasoning, and planning. As shown below, the suggested baselines failed to complete any of the tasks in our benchmark with a high success rate.

**Ref:**

[1] Qin, Meiying, Jake Brawer, and Brian Scassellati. 2022. “Robot Tool Use: A Survey.” Frontiers in Robotics and AI 9: 1009488.

Reviewer comment (Experiments):
The ablations provided are interesting and welcome, but could the authors include some more well-established baselines such as [1] or [2] in this evaluation?

Author response:
> Additional baselines

 - Thanks for pointing us to the baselines. We compare our proposed RoboTool with Code as Policies (CaPs) and ViperGPT. To ensure a fair comparison, the prompts for the baselines contain all the information provided to RoboTool. Moreover, baselines additionally have access to APIs presented in their original papers. All the baselines and RoboTool are based on gpt-4. We show the success rates in the simulation in the following Table 1. The results show that RoboTool performs better than the two baselines. We also provide the error breakdown in Table 2 below. Our results show that RoboTool can successfully reduce the physical tool-use errors (which physical tool to use), logical errors (how to devise a plan), and numerical errors (what are the parameters for each robot skill). The results strongly support the contributions of RoboTool.

Table 1: Success rates of RoboTool and additional baselines.
|  Task | RoboTool (ours) | CaPs | ViperGPT |
| --- | -------- | -------------- | -------- |
|  Milk-Reaching  | 0.9 | 0.0 | 0.0 |
|  Can-Grasping  | 0.7 | 0.2 | 0.0 |
|  Button-Pressing  | 0.8 | 0.6 | 0.6 |
|  Sofa-Traversing  | 1.0 | 0.4 | 0.3 |
|  Sofa-Climbing  | 1.0 | 0.4 | 0.2 |
|  Cube-LIfting  | 0.6 | 0.4 | 0.0 |
|  Average  | 0.83 | 0.33 | 0.18 |

Table 2: Error beakdowns for the additional baselines.
|     | RoboTool | CaPs | ViperGPT |
| --- | ---------------- | ------------ | -------------------- |
Tool Use Error| 0.00 |  0.23| 0.38|
Logical Error| 0.09 |  0.22 | 0.30|
Numerical Error| 0.08|  0.22 | 0.13|
Success| 0.83| 0.33| 0.19 |

Reviewer comment (Novelty):
Can the authors address why an API-only algorithm is sufficiently novel? In particular, I don't see where there is any learning of representations, which nominally is what ICLR is focused on.

Author response:
> ICLR is focused on representation learning, while RoboTool is a training-free method.

 - Despite the name, ICLR is about more than just learning representation, as documented by the non-comprehensive topic list on the [official website](https://iclr.cc/Conferences/2024/CallForPapers). This year in ICLR, there are at least 90 submissions related to training-free LLM Agents. When looking at the accepted papers of this year NeurIPS, one can find at least 20 impactful training-free API-based LLM agents, such as HuggingGPT [2], Reflexion [3], DEPS [4], AVIS [5], CAAFE [6], and SheetCopilot [7]. With increasingly capable LLMs, it is critical to explore what they can or cannot do, which is underexplored especially in the physical interaction and 3D reasoning domains. RoboTool is one of the first steps towards reasoning implicit environment- and embodiment-constraints and long-horizon hybrid discrete-continuous planning. Therefore, RoboTool is well-suited to the top AI publication venues.

**Ref:**

[2] Shen, Yongliang, et al. "Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface." NeurIPS 2023.

[3] Shinn, Noah, Beck Labash, and Ashwin Gopinath. "Reflexion: an autonomous agent with dynamic memory and self-reflection." NeurIPS 2023.

[4] Wang, Zihao, et al. "Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents." NeurIPS 2023.

[5] Hu, Ziniu, et al. "AVIS: Autonomous Visual Information Seeking with Large Language Models." NeurIPS 2023.

[6] Hollmann, Noah, Samuel Müller, and Frank Hutter. "GPT for Semi-Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering." NeurIPS 2023.

[7] Li, Hongxin, et al. "SheetCopilot: Bringing Software Productivity to the Next Level through Large Language Models." arXiv preprint arXiv:2305.19308 (2023).

Reviewer comment (Writing):
While the aim proposed by the authors is “we aim to solve a hybrid discrete-continuous planning problem”, this is not solved in this work or at least not described properly.

Author response:
Thanks for the question. We explain the hybrid task-motion planning problem here in more detail. An important part of TAMP problems is the interdependence of the motion-level and task-level aspects of the problem. Ignoring such interdependence between them will make it unable to solve the problem. We borrow this example from the seminal TAMP survey [1]:
 - Consider a task where the robot needs to place a particular object (name $A$) at a location. It might select a high-level plan "skeleton" such as "moveF$(q_0, \tau_1, q_1, p_0)$, pick[$A$]$(q_1, p_0, g)$, moveH[$A$]$(g, q_1, \tau_2, q_2)$, place[$A$]$(q_2, p_1, g)$", where 
     - moveF action involves robot movement when the gripper has no object with it.
     - moveH[$A$] action involves robot movement when the gripper is holding object $A$.
 - The action primitives are discrete options. However, the free parameters are continuous values, such as robot configurations $(q_0, q_1, q_2)$, a grasp pose $(g)$, placement poses $(p_0, p_1)$, paths $(\tau_1, \tau_2)$. The skeleton imposes constraints on the choices of those values, or conversely, there could be no satisfying set of values, and the skeleton needs to be changed. For example, if there is object $B$ occupying the target location, the robot needs to first remove $B$ and then place $A$ at the target location.

This example showcased the importance of hybrid discrete-continuous planning. We will modify the problem formulation accordingly in the final version to highlight this.

**Ref:**

[1] Garrett, Caelan Reed, et al. "Integrated task and motion planning." Annual review of control, robotics, and autonomous systems 4 (2021): 265-293.

Reviewer comment (Novelty):
The clarity of what is achieved should be improved. The first contribution: “long-horizon hybrid discrete-continuous planning” is not solving the hybrid part, as it is using predefined skills.

Author response:
Thanks for the question. We explain the hybrid task-motion planning problem here in more detail. An important part of TAMP problems is the interdependence of the motion-level and task-level aspects of the problem. Ignoring such interdependence between them will make it unable to solve the problem. We borrow this example from the seminal TAMP survey [1]:
 - Consider a task where the robot needs to place a particular object (name $A$) at a location. It might select a high-level plan "skeleton" such as "moveF$(q_0, \tau_1, q_1, p_0)$, pick[$A$]$(q_1, p_0, g)$, moveH[$A$]$(g, q_1, \tau_2, q_2)$, place[$A$]$(q_2, p_1, g)$", where 
     - moveF action involves robot movement when the gripper has no object with it.
     - moveH[$A$] action involves robot movement when the gripper is holding object $A$.
 - The action primitives are discrete options. However, the free parameters are continuous values, such as robot configurations $(q_0, q_1, q_2)$, a grasp pose $(g)$, placement poses $(p_0, p_1)$, paths $(\tau_1, \tau_2)$. The skeleton imposes constraints on the choices of those values, or conversely, there could be no satisfying set of values, and the skeleton needs to be changed. For example, if there is object $B$ occupying the target location, the robot needs to first remove $B$ and then place $A$ at the target location.

This example showcased the importance of hybrid discrete-continuous planning. We will modify the problem formulation accordingly in the final version to highlight this.

**Ref:**

[1] Garrett, Caelan Reed, et al. "Integrated task and motion planning." Annual review of control, robotics, and autonomous systems 4 (2021): 265-293.

Reviewer comment (Novelty):
For completeness, the reviewer is missing the LLM approach to Robotics: PaLM-E: An Embodied Multimodal Language Model.

Author response:
- Thanks to the reviewer for pointing out these two papers. We added them into the additional related work in the appendix and will incorperate them to the final version of related work.

Reviewer comment (Novelty):
The reviewer is also missing recent works on planning with low-level control such as: Active inference and behavior trees for reactive action planning and execution in robotics, TRO 2023.

Author response:
- Thanks to the reviewer for pointing out these two papers. We added them into the additional related work in the appendix and will incorperate them to the final version of related work.

Reviewer comment (Experiments):
For a fair baseline comparison, authors can use a PDDL planner as baseline. It may be misleading to call a baseline comparison an ablation study of the own algorithm.

Author response:
- **Compare with a PDDL algorithm**: Although we formulated creative tool use as a planning problem, solving such tasks requires more than just geometric planning. PDDL and other formal-logic planners, for example [2], are able to solve the Can-Grasping task, which requires only geometric planning. However, they cannot solve some other tasks in this work, for example, Cube-Lifting, which requires identifying a hidden lever structure in the scene and finding a way to activate this mechanism. The traditional planners are not able to reason based on common knowledge or manufacture nonexistent tools, which are critical to solving other creative tool-use tasks. 

**Ref:**

[2] Toussaint, Marc, Kelsey Allen, Kevin Smith, and Joshua Tenenbaum. 2018. “Differentiable Physics and Stable Modes for Tool-Use and Manipulation Planning.” In Robotics: Science and Systems XIV. Robotics: Science and Systems Foundation. https://doi.org/10.15607/rss.2018.xiv.044.

Reviewer comment (Experiments):
It is not clear the type of randomization in the environment initialization to properly evaluate the accuracy of the planner.

Author response:
- **Randomization:** For robot arm experiments, we randomize the initial configuration of objects. For quadrupedal robot experiments, we randomize the initial position of movable objects and the robots. We show the randomization range in Table 1 and add it to Appendix G in the updated draft.

Table 1: Randomization of objects in each task.
| | Objects: Randomization range [x_min, x_max, y_min, y_max] |
| ------------- | ------- |
| Milk-Reaching | pineapple toy, lock, cube, tomato toy: [0, 0.45, -0.25, 0.25]; hammer: [0, 0.45, -0.25, 0]; milk: [0.45, 0.6, 0, 0.25]  |
| Can-Grasping | stick: [0, 0.45, -0.25, 0]; paper: [0, 0.45, 0, 0.25]; can: [0.45, 0.7, -0.1, 0.1]  |
| Button-Pressing | magnetic_cube1, magnetic_cube2, magnetic_cube3: [0, 0.1, -0.25, 0.25]; button: [0.45, 0.7, -0.25, 0.25]  |
| Sofa-Traversing | surfboard: [0.3, 0.6, -0.175, 0.0]; cloth: [0.3, 0.6, 0.0, 0.2]; robot: [-0.5, -0.2, -0.2, 0.2];   |
| Sofa-Climbing | box_2: [0.0, 2.0, 0.0, 1.0]; robot: [0.0, 3.0, 0.0, 2.0];   |
| Cube-Lifting | chair: [0.8, 1.2, -0.1, 0.1]; surfboard: [0.8, 1.2, 0.3, 0.5]; robot: [0.0, 3.0, 0.0, 2.0];   |

Reviewer comment (Reproducibility):
There is no mention on what type of LLM is being used and how it is pretrained and refined for each component; the reviewer considers this important information.

Author response:
- In the introduction on page 2, we mentioned that "All of these components are constructed using GPT-4." We will highlight it again in the experiment part.

Reviewer comment (Evaluation):
In “Hierarchical Policies for Robot Tool Use” there is no analysis of the combinatorial nature of the parametrized skills. The reviewer asks: we are talking about 4 skills with how many parameters? How many instances of objects? This is important to understand the level of complexity of the decision tree.

Author response:
- For the robotic arm, we have a skill library that includes```[get_position, get_size, open_gripper, close_gripper, move_to_position```. Among them, ```get_position``` and ```get_size``` take in discrete ```object_name``` as input, while ```move_to_position``` takes in 3D position of the target gripper as input. The longest plan (Can-Grasping) requires about 15 steps. A more detailed description of the parameterized skills can be found in Appendix B in the original draft.

Reviewer comment (Experiments):
The reviewer questions whether 4 LLMs are needed to solve simple reasoning and generate a plan.

Author response:
- Thanks for the insightful question. When developing RoboTool, we noticed that LLMs are better at handling homogenous tasks rather than heterogeneous ones. Such an observation has also been discovered by prior literature [3]. To get a quantitative comparison, we conducted experiments by concatenating all the prompts of different modules in RoboTool and calling got-4.0 at once. We get a final success rate of 0.03, which is much worse than the proposed RoboTool.

**Ref:**

[3] Yu, Wenhao, et al. "Language to Rewards for Robotic Skill Synthesis." arXiv preprint arXiv:2306.08647 (2023).

Reviewer comment (Theory):
Humans have mechanical/dynamics knowledge learnt from experience. Although the authors mention affordances, it is not only about semantics but also about the real interaction in the environment.

Author response:
- Thank you for the insightful comment. We agree with the reviewer about how affordances related to physics properties in robotics. Indeed, physics-grounded knowledge is key to robust interaction with the environment. Recent work has started to investigate this direction, such as physically grounded Vision-Language Models [4]. We also believe that the real-world interactions would enable a more robust embodied LLM system, which would be an interesting future direction for RoboTool.
 - In this work, "creative" refers to using tools beyond their designated or default functionality. As discussed above, we think exploration with physical interactions might also lead to more interesting, diverse, and creative solutions. We appreciate the reviewer for the insightful comments.

**Ref:**

[4] Gao, Jensen, et al. "Physically Grounded Vision-Language Models for Robotic Manipulation." arXiv preprint arXiv:2309.02561 (2023).

Reviewer comment (Evaluation):
There is no analysis of how the system handles uncertainty resolution and trades off exploitation vs exploration (or intrinsic motivation). The reviewer notes this is important for creativity; reasoning alone is not enough to induce creativity, but reasoning combined with uncertainty resolution to try new things could lead to artificial creativity.

Author response:
- Thank you for the insightful comment. We agree with the reviewer about how affordances related to physics properties in robotics. Indeed, physics-grounded knowledge is key to robust interaction with the environment. Recent work has started to investigate this direction, such as physically grounded Vision-Language Models [4]. We also believe that the real-world interactions would enable a more robust embodied LLM system, which would be an interesting future direction for RoboTool.
 - In this work, "creative" refers to using tools beyond their designated or default functionality. As discussed above, we think exploration with physical interactions might also lead to more interesting, diverse, and creative solutions. We appreciate the reviewer for the insightful comments.

**Ref:**

[4] Gao, Jensen, et al. "Physically Grounded Vision-Language Models for Robotic Manipulation." arXiv preprint arXiv:2309.02561 (2023).

Reviewer comment (Experiments):
A lot of engineering seems to have gone into these examples. The prompts (on separate github) contain a number of hints on what not to do when solving the problem, which seem engineered for the particular tasks. Examples: "If you do not know the actual value, use an offset = 1m."; "You must be careful when calculating with negative values."; "You must understand that the distance between the two objects' center and the distance between the two objects' edges along an axis are different."; The coder in Fig.2 also generates a seemingly arbitrary gripping offset for the hammer which I guess you engineered depending on the shape of the hammer. Some motion primitives are a bit contrived: As a roboticist, getting the robot to kick the surfboard in place to traverse the sofa seems like an extremely challenging task that I guess you just spent a lot of time engineering the motion primitives for. I don't think these are very realistic examples of your approach considering how much engineering must have gone into them. It is very difficult to say how much going on here is just simple symbolic task planning vs. motion planning (e.g., the real-valued positions and orientation parameters).

Author response:
>  The prompts contain hints on what not to do when solving the problem.

 - We removed the prompt related to "offset=1.0" and "careful when calculating negative values." After rerunning the experiments, we found some decrease in RoboTool's success rate (0.87 -> 0.83); however, it was insignificant. We updated Table 1 in the draft accordingly.
 - The prompts related to the negative values and distances aim to remedy LLMs' deficiency, which hardly hints at how to plan for a specific task. Please note that we use the same prompts for all experiments, not specifically engineered for each task. Designing prompts containing general rules to follow is common in many embodied LLMs literature. 
     - Regarding the prompt mentioning that the distance between objects' centers and boundaries are different, it is related to the 3D reasoning capability of gpt-4: It sometimes mistakenly thinks that the distance between objects is the distance between their centers without considering the size of the objects. We want to reduce such mistakes by reinforcing the distance concept in 3D world. However, RoboTool decides whether to use the distance between centers or the distance between boundaries for downstream planning.

> Is the gripping offset in Fig. 2 code for the hammer engineered based on the shape of the hammer?

 - The numbers are not engineered based on the shape of the hammer. The calculator prompts contain an example of calculating a generic object's grasping position. In other generated codes, RoboTool follows the examples in the prompt and automatically gets the grasping offset based on the object's bounding box size. We provide more generated codes in Appendix I.
 - We need such an example in the prompt since RoboTool uses general motion primitives, such as ```move_to_position(pos)```  and ```close_gripper()``` to provide extra planning flexibility, instead of more granular primitives, such as ```pick(object)``` and ```place(object)``` in many existing methods [1, 2, 3, 4]. RoboTool needs to calculate an accurate gripper target position, which is difficult to rely solely on the prior knowledge of LLMs alone.

**Ref:**

[1] Singh, Ishika, et al. "Progprompt: Generating situated robot task plans using large language models." 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023.

[2] Ahn, Michael, et al. "Do as i can, not as i say: Grounding language in robotic affordances." arXiv preprint arXiv:2204.01691, 2022.

[3] Huang, Wenlong, et al. "Grounded decoding: Guiding text generation with grounded models for robot control." arXiv preprint arXiv:2303.00855, 2023c.

[4] Liang, Jacky, et al. "Code as policies: Language model programs for embodied control." 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023.

> The engineering effort put in motion primitives.
 - Thanks for the question. Our ```push_to_position()``` skill is based on Unitree's built-in walking mode gait with a small gait height and a motion planner to generate waypoints to move objects around (as shown in Appendix B.1 in the original draft). There is surging interest in obtaining better loco-manipulation skills [6-8], which is manipulating objects with a quadrupedal robot. We provide one simple implementation of such loco-manipulation skill, which is already sufficient to perform complex long-horizon tasks. Although obtaining such motion primitive is not the focus of this work, we are eager to integrate more diverse skills in RoboTool. 
 - Instead, in this work, we focus on high-level task planning and assume access to a library of existing motion primitives, similar to SayCan [2], GroundedDecoding [3], PPDL with LLMs [5] etc. Please note that our method is general and is compatible with skills derived from different methods, such as classical planning, RL, or imitation learning. 


**Ref:**

[5] Silver, Tom, et al. "Generalized planning in pddl domains with pretrained large language models."
arXiv preprint arXiv:2305.11014, 2023.

[6] Nachum, Ofir, et al. "Multi-agent manipulation via locomotion using hierarchical sim2real." arXiv preprint arXiv:1908.05224 (2019).

[7] A. Rigo, Y. Chen, S. K. Gupta, and Q. Nguyen, “Contact optimization for non-prehensile loco-manipulation via hierarchical model predictive control,” arXiv preprint arXiv:2210.03442, 2022.

[8] M. Sombolestan and Q. Nguyen, “Hierarchical adaptive locomanipulation control for quadruped robots,” arXiv preprint arXiv:2209.13145, 2022.

Reviewer comment (Experiments):
It relies only on ChatGPT 4.0 which means that it is unclear to me if this architecture design and ablation study would generalize to other LLMs. GPT-4.0 is much better than open source models so the decision is understandable but it is a weakness of the paper. It also makes reproducing it harder since ChatGPT is updated and has been observed to change behavior over time (OTOH it is very easy to use the current version of GPT4...).

Author response:
> Can RoboTool generalize to other LLMs?
 - RoboTool is compatible with different LLMs. We provide additional results using both gpt-3.5-turbo and gpt-4. We compare RoboTool with Code as Polices (CaPs) showing the per-task success rates in Table 1 and error breakdowns in Table 2. Table 1 shows that when using different language models, RoboTool could consistently outperform CaPs, showing the generalizability of RoboTool.
 - Based on the error breakdowns in Table 2, RoboTool helps reduce the tool use error (whether the correct physical tool is selected) when using gpt-3.5-turbo. However, gpt-3.5-turbo cannot devise a reasonable plan with the selected physical tool. Those observations highlight the tremendous challenges in reasoning, planning, and calculation for creative robot tool use tasks. Such demanding tasks left us with very few options for competent LLMs. 
 - As suggested by the reviewers, open-sourced LLMs are generally preferable for academic research. However, as in many concurrent works, gpt-4 is found to be unequivocally the best-performing LLM across different tasks and embodiments. We will keep a close look at the open-sourced models and update RoboTool once they achieve performance comparable to gpt-4. 

Table 1: Additional Baselines.
|                 | RoboTool (gpt-4) | RoboTool (gpt-3.5-turbo) | CaPs (gpt-4) | CaPs (gpt-3.5-turbo) |
| --------------- | ---------------- | ------------------------ | ---------------------- | ------------------------------ |
| Milk-Reaching   | 0.9              | 0.0                      | 0.0                    | 0.0                            |
| Can-Grasping    | 0.7              | 0.0                      | 0.2                    | 0.0                            |
| Button-Pressing | 0.8              | 0.2                      | 0.6                    | 0.0                            |
| Sofa-Traversing | 1.0              | 0.2                      | 0.4                    | 0.0                            |
| Sofa-Climbing   | 1.0              | 0.0                      | 0.4                    | 0.0                            |
| Cube-LIfting    | 0.6              | 0.0                      | 0.4                    | 0.0                            |
| Average         | 0.83             | 0.07                     | 0.33                   | 0.00                            |

Table 2: Error breakdowns for new baselines.
|     | RoboTool (gpt-4) | RoboTool (gpt-3.5-turbo) | CaPs (gpt-4) | CaPs (gpt-3.5-turbo) |
| --- | ---------------- | ------------------------ | ------------ | -------------------- |
Tool Use Error| 0.00 | 0.45 | 0.23| 0.96|
Logical Error| 0.09 | 0.41 | 0.22 | 0.02|
Numerical Error| 0.08| 0.07 | 0.22 | 0.02|
Success| 0.83| 0.07 | 0.33| 0.00 |

Reviewer comment (Novelty):
Relation to other LLM works that do manipulation could maybe be clarified, e.g., VoxPose was kind of brushed off as being multi-modal, but isn't that a strength? IIRC they also show somewhat complex actions (grabbing a toast and putting it on a cutting board). Your example is more complex but it is difficult to quantify since they seem heavily engineered.

Author response:
> Comparison with VoxPoser
 - Thanks for the question. RoboTool and VoxPoser have different focuses and are complementary. VoxPoser does not focus on complex reasoning or long-horizon task plans like RoboTool. VoxPoser aims to generate low-level trajectories given clear decomposed task instructions. One possible way to integrate the two methods is by using RoboTool to generate the task plan and VoxPoser to generate low-level trajectories.
 - Multi-modality is indeed a strength when it comes to LLM agents. Like VoxPoser, RoboTool interacts with [OWL-ViT](https://huggingface.co/docs/transformers/model_doc/owlvit) to detect various objects. As discussed in the limitation section, a promising future direction would be integrating the vision-language model seamlessly into RoboTool. In this paper, we first want to show the initial signs of solving creative tool-use tasks with only LLMs.

Reviewer comment (Reproducibility):
Why are the prompts in the appendix only links to github instead of actually in the appendix? IIRC there is no page limit on the ICLR appendix.

Author response:
> Minor issues with the prompt link and task descriptions
 - In Appendix D and E of the initial submission, we included hyperlinks that directly jump to the prompts and task descriptions, as we thought that is a cleaner presentation. We sincerely apologize if this choice caused inconvenience to the reviewers. As suggested, we have included the prompts in the updated version instead of links. 
 - Please note that the descriptions fed to the LLMs contain task descriptions following the template "You want to ...", such as "You want to grasp the milk" or "You want to get on top of the sofa". We apologize for the confusion and correct the descriptions in the introduction as pointed out by the reviewer ("grasping" -> "grasp", "walking" -> "walk").

Reviewer comment (Writing):
The language could be better, especially the examples in the intro do not actually seem to be grammatically correct calls for action, e.g., "grasping a milk cartoon" should be "grasp a milk cartoon", "walking to the sofa" should be "walk to the sofa" and so on. This is a bit problematic if a prompt is unintentionally ambiguous as it could affect your results.

Author response:
> Minor issues with the prompt link and task descriptions
 - In Appendix D and E of the initial submission, we included hyperlinks that directly jump to the prompts and task descriptions, as we thought that is a cleaner presentation. We sincerely apologize if this choice caused inconvenience to the reviewers. As suggested, we have included the prompts in the updated version instead of links. 
 - Please note that the descriptions fed to the LLMs contain task descriptions following the template "You want to ...", such as "You want to grasp the milk" or "You want to get on top of the sofa". We apologize for the confusion and correct the descriptions in the introduction as pointed out by the reviewer ("grasping" -> "grasp", "walking" -> "walk").

Reviewer comment (Experiments):
How much do you vary/randomize in your sim and real robot examples, does the robot and world always start in the same configuration?

Author response:
> Randomization in sim and real robot examples. 
 - Thanks for the question. We indeed randomized the configurations in experiments and sorry for the confusion. For robot arm experiments, we randomize the initial configuration of objects. In quadrupedal robot experiments, we randomize the initial positions of movable objects and the robots. We show the randomization ranges in Table 3 and add it to Appendix G in the updated draft.

Table 3: Randomization of objects in each task.
| | Objects: Randomization range [x_min, x_max, y_min, y_max] |
| ------------- | ------- |
| Milk-Reaching | pineapple toy, lock, cube, tomato toy: [0, 0.45, -0.25, 0.25]; hammer: [0, 0.45, -0.25, 0]; milk: [0.45, 0.6, 0, 0.25]  |
| Can-Grasping | stick: [0, 0.45, -0.25, 0]; paper: [0, 0.45, 0, 0.25]; can: [0.45, 0.7, -0.1, 0.1]  |
| Button-Pressing | magnetic_cube1, magnetic_cube2, magnetic_cube3: [0, 0.1, -0.25, 0.25]; button: [0.45, 0.7, -0.25, 0.25]  |
| Sofa-Traversing | surfboard: [0.3, 0.6, -0.175, 0.0]; cloth: [0.3, 0.6, 0.0, 0.2]; robot: [-0.5, -0.2, -0.2, 0.2];   |
| Sofa-Climbing | box_2: [0.0, 2.0, 0.0, 1.0]; robot: [0.0, 3.0, 0.0, 2.0]  |
| Cube-Lifting | chair: [0.8, 1.2, -0.1, 0.1]; surfboard: [0.8, 1.2, 0.3, 0.5]; robot: [0.0, 3.0, 0.0, 2.0]  |

Reviewer comment (Reproducibility):
Can you also say something about how much the action parameters are varied in response to these, or maybe show us some example python code from different runs of the algorithm on e.g., the milk example?

Author response:
> How much are the action parameters varied to the randomization? Maybe show us some example Python codes.

 - Please note that RoboTool usually gets the 3D positions of objects in the scene by invoking the ```get_position(object)``` primitive, and assigning it to a variable in the code for downstream computation with numpy. Therefore, the code may look very similar to each other, but the actual parameter values are different. Note that in Section 5.5, Discriminative Tool-use Capability, we tested RoboTool's ability to use tools when necessary and ignore tools when the robot can directly finish tasks without the need to manipulate other objects. That is when the plan and code look very different when given different task configurations.
 -  We provide more Python codes in Appendix I to show that RoboTool can vary parameters based on the environment.