Paper: LLM+A: Grounding Large Language Models in Physical World with Affordance Prompting (ICLR-2024)

Reviewer comment (Experiments):
[Limited Generalization]: The experiment of this study depends on the simplified tasks, such as pushing cubes and put cubes. These objects are rather simple with regular shapes. This is crucial to discretize the actions. However, the experiments may not cover all possible scenarios with more complex envs and irregular daily objects, leading to potential limitations in the model's applicability outside the tested conditions.

Author response:
Thank you for your valuable suggestions. We agree with you that the experiment should include more complex tasks to verify the generalization ability of our method. Accordingly, we have incorporated additional testing scenarios. First, we diversified the shape of objects, introducing cubes, pentagons, star-shaped, and moon-shaped objects. We conducted an evaluation of our method over 100 episodes in the Block2Position task, resulting in a success rate of 38%. Notably, this rate is nearly consistent with the success rate reported in our paper. An example is shown in Figure 5. Second, we applied our method in complex scenarios, involving potential interferences with object positions. More specifically, we randomly change the object position at any given time step within an episode. An example of environmental observation and robot trajectories is provided in Figure 6 for your reference. The results demonstrate the robustness of our method, notably its ability to dynamically recalibrate motion sequences.

Reviewer comment (Evaluation):
[Affordance Predictions Accuracy] The accuracy of affordance predictions from LLMs could be a potential weakness. If the affordance values are inaccurately predicted, it might lead to sub-optimal or erroneous robotic actions. Variability of landscapes, backgrounds, and language descriptions predicting affordances across different objects or environments could impact the overall performance.

Author response:
We agree with your concern about the affordance prediction accuracy. Variability of landscapes, backgrounds, and language descriptions across different objects or environments could impact the over-performance. In our experiments, the task instruction has no pre-defined template, grammar, or vocabulary for each task. For instance, “move the red block to the top right of the board” can also be described as “push the red block to the upper right corner”. The experiment results show that LLM is robust to the variability of language descriptions. 

Besides, the robustness to the variability of landscapes and backgrounds is mainly dependent on the capability of detection models, recent visual-language models can achieve pleasant detection/segmentation results with respect to text inputs, and the accuracy of the perception module is not the focus of our work.

Furthermore, even if the affordance prediction can make mistakes sometimes, as our method takes in the detection results as the planner’s input which composes the closed-loop control, it can greatly alleviate the planning prediction error in one round by consistently adjusting its planning trajectory.

Reviewer comment (Experiments):
Overall, the experiments might lack certain real-world complexities, such as dynamic and unpredictable environments. For example, the Owl-vit vision grounding module only predicts the bounding box, which is not ideal for most of the cases in real-world applications.

Author response:
Thank you for your comments, and I agree with you that real-world experiments are the best way to illustrate LLM+A’s ability in robotics control. And we will append the real-world experiments in our future work. But in principle, as our method works in a zero-shot manner, it is less likely to suffer from the sim-to-real transfer. First of all, the inputs of our LLM+A system only include the bounding box of the objects. Recent advances in general visual grounding modules/visual-language models can achieve satisfying accuracy in predicting the bounding box/segmentation mask of the objects, even for object parts [1][2][3]. The RGB-D camera is commonly used in robotics manipulation and grasping fields. With the help of the depth sensor, it is straightforward to translate the planned waypoints from pixel space into a 3-D coordinates in the real world, and then use IK to control the locomotion of the robot arm. I understand your concern about the generalization of the bounding-box-based inputs. However, our framework focuses on the “parts” of the objects instead of being limited in bounding boxes. In our experiments, the simulator provides tabletop manipulation tasks on blocks, so it is natural to use bounding boxes and edges to represent the “parts” of a block. But for real-world applications, an object often comes with functional parts, like the hammer is composed of a handle and a hammer head. It is not difficult for the LLM to first infer the parts of the objects and then use vision-language models to predict the location of different parts to serve as our LLM+A inputs. 

[1] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W., Dollár, P., & Girshick, R.B. (2023). Segment Anything. ArXiv, abs/2304.02643.

[2] Lai, X., Tian, Z., Chen, Y., Li, Y., Yuan, Y., Liu, S., & Jia, J. (2023). LISA: Reasoning Segmentation via Large Language Model. ArXiv, abs/2308.00692.

[3] Wang, W., Chen, Z., Chen, X., Wu, J., Zhu, X., Zeng, G., Luo, P., Lu, T., Zhou, J., Qiao, Y., & Dai, J. (2023). VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks. ArXiv, abs/2305.11175.

Reviewer comment (Evaluation):
What is the inference speed of the planning? Prompting GPT-4 takes some time to response compared with other simpler models e.g. from Ros integration.

Author response:
To increase the time efficiency, we query GPT4 to in a fixed interval generate sub-task plans and motion sequences instead of re-planning new trajectory waypoints every time step. Specifically, the planned sequence will be updated only after the robot finishes five waypoints in the previous query round. The average query number of each episode in evaluation tasks can be referred to in the following table. We do not include the query time since it depends on different sources of OpenAI APIs.
| Task | Block2Position | Block2Block | Separate |
| --- | --- | --- | ---|
| GPT4 average query number | 3.86 | 3.75 | 1.82 |

Reviewer comment (Experiments):
Why not implement SAM based vision detection modules for more accurate detection and generalize the tasks into more complex scenarios.

Author response:
Thank you for your advice. SAM can provide a more detailed description of the object's shape. However such detailed information is difficult to translate into text-inputs for LLM to understand. Maybe the SAM mask input is more suitable for  VLM (like recent GPT-4V) to do the plan. We use the bounding-box detection inputs as a trade-off between the description details and the difficulty of translating into texts. One major disadvantage of using the bounding box is that the object will not occupy all the space indicated by the bounding box. The planner may plan a waypoint that does not actually lead to interactions between the robot arm and the object. But as our LLM+A uses the detection results as feedback, it will be aware of such mistakes and improve its previously planned trajectory to fix this.

Reviewer comment (Experiments):
The paper title claims physical world, however, in the evaluation, it only considers simulation tasks on table top (2D). Physical world robotics interaction is much more complicated than simulation and will absolutely break the assumption of this paper. In my point of view, the technique proposed in this paper only applies to a very limited setup. Basically, given some 2D points (target positions) how to use robot arm (source positions) to reach it and generate some trajectories. In contrary, techniques such as code as policies is general and can extend to physical world.

Author response:
Thank you for your comments. In [1], the simulated environment “Language-Table” consists of a simulated 6DoF robot xArm6 implemented in PyBullet equipped with a small cylindrical end effector. Third-person perspective RGB-only images from a simulated camera are used as visual input. Also, similar real-world experiments are tested using UFACTORY xArm6 robot arms. The authors have demonstrated that the policy performance in Language-Table was highly correlated with policy performance in the real world. In [2], the Pick&Place task is built based on the Ravens benchmark [3] set in PyBullet. Although the rendering software stack may not fully reflect the noise characteristics often found in real data, the simulated environment can roughly match the real-world setup. Despite the 2D environment, we can transfer our method into a 3D setup using the following pipeline: we first invoke open-vocab detector OWL-ViT [4] to obtain a bounding box, and feed it into Segment Anything [5] to obtain an object mask. Then the object point cloud can be reconstructed with the mask and the RGB-D observation.

[1] Lynch, Corey, et al. "Interactive language: Talking to robots in real time." IEEE Robotics and Automation Letters (2023).

[2] Shridhar, Mohit, Lucas Manuelli, and Dieter Fox. "Cliport: What and where pathways for robotic manipulation." Conference on Robot Learning. PMLR, 2022.

[3] Zeng, Andy, et al. "Transporter networks: Rearranging the visual world for robotic manipulation." Conference on Robot Learning. PMLR, 2021.

[4] Minderer, M., et al. "Simple open-vocabulary object detection with vision transformers”. arXiv 2022. arXiv preprint arXiv:2205.06230.

[5] Kirillov, Alexander, et al. "Segment anything." arXiv preprint arXiv:2304.02643 (2023).

Reviewer comment (Experiments):
The technique proposed by this paper may highly depend on the choice of LLM. Ablations w/ different LLMs is required to show generality.

Author response:
Thank you for your valuable suggestion. Our method currently still depends on the powerful GPT4. GPT4 is prone to generate outputs as we instructed that can be executed in the code, which is necessary for the manipulator policy. However, open-source models like Llama2 or Vicuna often throw errors both syntactically and semantically even if we show exemplars and instruct the output format in the prompt. Maybe instruction finetuning can alleviate this issue, but this is not the focus of our paper.

Reviewer comment (Experiments):
In SayCan paper, there is an "open source environment" section. Looks like the tasks are similar. It will be interesting to see the comparison w/ SayCan.

Author response:
Thank you for your comments. The work SayCan aims to provide real-world grounding by means of pre-trained skills. In comparison, our work utilizes LLMs as both high-level task planners and low-level motion controllers without dependency on pre-trained skills. Therefore, it may be unfair to include SayCan as the baseline.

Reviewer comment (Experiments):
Is the policy able to re-try if the first trial fails?

Author response:
Yes. We query the LLM to re-plan sub-tasks and motion sequences every five time steps to correct any possible fails. To approve the effectiveness of re-plan, we applied our method in complex scenarios, involving potential interferences with object positions. More specifically, we randomly change the object position at any given time step within an episode. An example of environmental observation and robot trajectories is provided in Figure 6 for your reference. The results demonstrate the robustness of our method, notably its ability to dynamically recalibrate motion sequences.

Reviewer comment (Novelty):
Limited technical contribution: this work comes off as an empirical evaluation of certain style of prompt engineering for robotic manipulation. Other than the fact that some prompt worked for a small set of robotic manipulation tasks, I am not sure what I learnt from this paper.

Author response:
In this work, we demonstrate that a pre-trained LLM (GPT4), equipped with only an off-the-shelf detection model, can guide a robot manipulator by outputting high-level sub-task plans and low-level sequences of end-effector waypoints. This is quite a challenging task. First, the proposed method does not depend on pre-trained skills, motion primitives, or trajectory optimizers. Second, the proposed prompting method is task-agnostic so can easily generalize to heterogeneous tasks. Third, since LLMs are not trained for grounded physical interaction, the proposed affordance prompting effectively improves the executability of both the sub-task plans and the control sequences. We appreciate that you denote that the number of robotic manipulation tasks is limited in the previous manuscript. In response, we plan to conduct more realistic robotic manipulation tasks in revision, such as “open the drawer”, “push the button”, etc.

Reviewer comment (Experiments):
The evaluation tasks are too simple. It is unclear how affordance prompting would scale to more complex or more real world tasks, for instance manipulation in cluttered settings or situations with partial observability.

Author response:
Thank you for your valuable suggestions and we agree with you that the experiment should include more complex tasks. Accordingly, we have incorporated additional testing scenarios. First, we diversified the shape of objects, introducing cubes, pentagons, star-shaped, and moon-shaped objects. We conducted an evaluation of our method over 100 episodes in the Block2Position task, resulting in a success rate of 38%. Notably, this rate is nearly consistent with the success rate reported in our paper. Besides, an example is shown in Figure 5. Second, we applied our method in complex scenarios, involving potential interferences with object positions. More specifically, we randomly change the object position at any given time step within an episode. An example of environmental observation and robot trajectories is provided in Figure 6 for your reference. The results demonstrate the robustness of our method, notably its ability to dynamically recalibrate motion sequences. Additionally, although partial observability may occur in real-world tasks, our method can address this issue through re-planning in the fixed interval.

Reviewer comment (Experiments):
It is unclear how well LLM+A will do without a really strong LLM such as GPT4. The use of GPT4 also makes it a computationally slow framework for online deployment. To that end, it would be good for the authors to report execution time/time to solve for their evaluation tasks. The reviewer also encourages the authors to do real-world evaluation to really put the runtime in perspective, and to see LLM+A performance with other open-source models like Llama2 or Vicuna.

Author response:
Thank you for your valuable suggestion.As the currently most powerful LLM, GPT4 encompasses plentiful commonsense and excellent abilities of reasoning and instruction following. Additionally, GPT is prone to generate outputs as we instructed that can be executed in the code. However, open-source models like Llama2 or Vicuna often throw errors both syntactically and semantically even if we show examples and instruct the output format in the prompt. Maybe instruction finetuning can alleviate this issue, but this is not the focus of our paper.

To increase the time efficiency, we query GPT4 to in a fixed interval generate sub-task plans and motion sequences instead of re-planning new trajectory waypoints every time step. Specifically, the planned sequence will be updated only after the robot finishes five waypoints in the previous query round. The average query number of each episode in evaluation tasks can be referred to in the following table. We do not include the query time since it depends on different sources of OpenAI APIs.
| Task | Block2Position | Block2Block | Separate | 
| --- | --- | --- | --- | 
| GPT4 average query number | 3.86 | 3.75 | 1.82 |

Reviewer comment (Experiments):
It is unclear how the authors chose the baselines they compare with. They do not provide a rationale on their selection of baselines. Instead they simply choose some subset of LLM prompt-based approaches. The reviewer asks if the goal was just to compare their prompt style, and why the authors did not also compare with Palm-e, RT-2, GATO, VIMA to show that their training-free prompt-based approach works better than these others that required additional data for training. Even if the goal is to compare prompt-based approaches, many others such as text2motion (https://sites.google.com/stanford.edu/text2motion) come to mind.

Author response:
We appreciate your concern regarding the selection of baselines in our study. Firstly, we would like to clarify that the choice of baselines was primarily limited by the availability of open-source code for replication. Unfortunately, many training-based methods such as Palm-e, RT-2, and GATO are proprietary and not openly accessible, which made it challenging for us to replicate and compare with these methods. Despite these limitations, we acknowledge that these training-based methods might possess an advantage in terms of final performance. However, our approach does not require training samples and demonstrates superior generalization capabilities, which makes it more accessible and easier to implement in diverse tasks. Therefore, we choose several training-free prompt-based methods for comparison. Regarding your suggestion of including text2motion as a baseline, we agree that it would be beneficial to include text2motion as a baseline for comparison. We will include the experimental results in revision..

Reviewer comment (Experiments):
Why not give examples to naive LLM baseline given that it needs to output coordinates in specific format too, just like LLM+A?

Author response:
Thank you for your suggestion. We have included an example of naive LLM in the Block2Postion task as shown in Listing 8. We found that Naive LLM makes many mistakes in spatial relationship reasoning. Specifically, naive LLM first fails in choosing the skills and proposes the wrong target coordinates in the subsequent step.

Reviewer comment (Experiments):
Do all other baselines also use GPT4? This is not mentioned anywhere in the paper, but the reviewer assumed this was the case. Please clarify.

Author response:
Thank you for your comment. We confirm that all other baselines also use GPT4 and this information is clarified in the revised manuscript.

Reviewer comment (Reproducibility):
The description of the Motion Controller could be improved as it is different in the experiments.

Author response:
In the motion controller, given the decomposed sub-tasks and the affordance values from the sub-task planner, the executable motion sequences for the robots are generated. This is consistent among different tasks in our experiments. We will clarify the related descriptions to avoid possible misunderstandings.

Reviewer comment (Experiments):
The work is relegating too much emphasis on affordance prompting, but this prompting is overengineered. For instance, Pick&place example does not shows enough the advantage of affordance prompting.

Author response:
We aim to explore the adaptation of the proposed affordance prompting technique. Therefore,  we do not further engineer the prompting style for each experimental task, which is verified to be effective in Pick&Place task. To further show the robustness of our method, we have incorporated additional testing scenarios. First, we diversified the shape of objects, introducing cubes, pentagons, star-shaped, and moon-shaped objects. We conducted an evaluation of our method over 100 episodes in the Block2Position task, resulting in a success rate of 38%. Notably, this rate is nearly consistent with the success rate reported in our paper. Besides, an example is shown in Figure 5. Second, we applied our method in complex scenarios, involving potential interferences with object positions. More specifically, we randomly change the object position at any given time step within an episode. An example of environmental observation and robot trajectories is provided in Figure 6 for your reference. The results demonstrate the ability of our method to dynamically recalibrate motion sequences. Additionally, we plan to conduct more realistic robotic manipulation tasks in the final version, such as “open the drawer”, “push the button”, etc.

Reviewer comment (Evaluation):
Results accuracy is very low comparing to other LLMs methods to solve planning.

Author response:
Most of the previous planning-based LLM methods (such as “Language Table”) depend on either pre-trained low-level skills or massive multi-modal data to encompass diverse robotic tasks. In contrast, we utilize LLM as both the high-level sub-task planner and the low-level motion controller without the need for additional training process/data which is advantageous in many real-world applications (e.g., the scenario where the agent needs to fast adapt to heterogeneous tasks). We acknowledge that accuracy is a current shortcoming of our method, but we believe the performance of such methods will improve along with the improvement of prompt design and the base LLM.

Reviewer comment (Experiments):
The example used for explaining why current methods do not work: “It may move directly right and then push the block” is not enough demonstrated with SOTA algorithms and maybe too biased by comparing with ReAct. For instance, Driess et al. PALM-E and interactive language can solve these type of table top problems.

Author response:
We would like to explain that the proposed method can be effective in many robotic manipulation tasks except for push blocks. The affordance prompting aims to infer goal-conditioned affordance values, which indicate the executable priorities of different parts of interacted objects. For example, when instructed to “pick the pot on the table”, the robot can be guided to hold the handle of the pot with affordance prompting. We plan to report more experimental results to demonstrate this advantage.

Reviewer comment (Novelty):
The section on previous works on affordances and tools could be improved with additional relevant literature, such as:  
Jamone, L., Ugur, E., Cangelosi, A., Fadiga, L., Bernardino, A., Piater, J., & Santos-Victor, J. (2016). Affordances in psychology, neuroscience, and robotics: A survey. IEEE Transactions on Cognitive and Developmental Systems, 10(1), 4-25.  
Fang, K., Zhu, Y., Garg, A., Kurenkov, A., Mehta, V., Fei-Fei, L., & Savarese, S. (2020). Learning task-oriented grasping for tool manipulation from simulated self-supervision. The International Journal of Robotics Research, 39(2-3), 202-216.

Author response:
Thank you for your valuable suggestions. We have included the recommended references in the current manuscript.

Reviewer comment (Experiments):
It is not totally clear who is setting the object parts and how the affordance values are being generated, as this is totally different in the pushing and the pick&place experiments.

Author response:
We manually choose the object parts for different kinds of tasks. For the pushing task, we distinguish the four edges of the bounding box as object parts. For the Pick&Place task, we detect the center of the bounding box as object parts. Then, we prompt the LLMs to directly generate affordance values ranging from 0 to 1 to represent the usefulness of each object part in the sub-task planner, and an example is shown in Listing 1 in the Appendix.

Reviewer comment (Writing):
In relation to the term "training-free," does this mean zero-shot, considering that the LLMs are trained?

Author response:
In this paper, the zero-shot setting denotes that we do not include any example outputs in the affordance prompting for different robotic tasks. To avoid possible misunderstandings, we have revised the related descriptions in the manuscript.

Reviewer comment (Theory):
Should affordances as goal-conditioned values be generalized to non-goal-conditioned effects?

Author response:
Thank you for your valuable suggestion. We will extend the affordance prompting to generate both non-goal-conditioned values and goal-conditioned values in our future work, which could possibly avoid the manual design of object parts.