Paper: GenBot: Generative Simulation Empowers Automated Robotic Skill Learning at Scale (ICLR-2024)

Reviewer comment (Experiments):
The proposed method incorporates random sampling of task objects and the robot agent during the seeding stage, taking into account the requisite skills for the desired tasks. This consideration of required skills, or the lack thereof, may potentially enhance learning efficiency.

Author response:
Dear reviewer,

Thank you for your constructive feedback and questions! We really appreciate your effort in reviewing our work and it is definitely helpful in improving our paper quality.

We are actively working on the rebuttal and adding experiments to address your concerns. Meanwhile, we would like to ask for some further clarifications so that we can address your concerns more clearly. We are not sure if we fully understand the first point you raised in the weakness section:

> The proposed method incorporates random sampling of task objects and the robot agent during the seeding stage, taking into account the requisite skills for the desired tasks. This consideration of required skills, or the lack thereof, may potentially enhance learning efficiency;

The object sampling is for seeding the LLM to generate semantically meaningful tasks, considering the affordance of the object. Could you clarify a bit what do you mean by this sentence, and what specific weakness are you concerned with? Any further explanation would be much appreciated!

Thank you!
Authors

Reviewer comment (Experiments):
Employing LLMs to generate task proposals can result in incomplete task information. For instance, in the context of a task such as "bowl heating," the LLM may generate a description that overlooks crucial details, like setting the timer.

Author response:
## 2. Potential incomplete task information using LLMs
> Employing LLMs to generate task proposals can result in incomplete task information. For instance, in the context of a task such as "bowl heating," the LLM may generate a description that overlooks crucial details, like setting the timer;

We would like to argue that generating tasks considering these semantically meaningful details, as well as object affordances, is actually a clear advantage of using LLMs for task generation, compared to prior approaches that rely on heuristic-based procedural generation. Since LLMs have developed reasonable common sense capabilities, they are able to produce these crucial low-level steps, while also providing natural language descriptions of them. In fact, in both Figure 1 and 2, we show that for the heating task, the generate task incorporates the crucial step for setting the timer in both task decomposition and language description.

If you are referring to necessary fine-grained details regarding the exact angle of the knob to be turned, we acknowledge that due to the lack of information of the knob, there's no pre-defined mapping from the turning degree to the exact time set in our current asset data. In these cases, our pipeline self-generates reasonable scale of the timer and then correspondingly the amount to turn during each generation run. Such generation also allows the data generated to cover a range of reasonable mapping scales. As long as such scale information is given at the time of deployment, a distilled policy will be able to condition on such extra information. Also, we would like to point out that other similar tasks such as "turning the clock by 1-hour", or "siding the window half-way" will produce accurate information, since the LLM encodes the common sense knowledge of computing the extent of "half-way" given a window length, and what angle corresponds to 1-hour. In addition, tasks with such fine-grained details has rarely been presented in previous robotic demonstration datasets (such as Metaworld or RLBench), as these prior works mostly consider changing the coarse spatial relationship between objects in the scene.

Reviewer comment (Reproducibility):
The generation of LLM-based reward functions heavily hinges on in-context prompts, which are derived from human comprehension of the task. This approach may necessitate significant human input and potentially restrict its applicability to novel tasks and domains.

Author response:
## 3. Requires human input
> The generation of LLM-based reward functions heavily hinges on in-context prompts, which are derived from human comprehension of the task. This approach may necessitate significant human input and potentially restrict its applicability to novel tasks and domains.


**Regarding human input**: we would like to clarify that our prompts, though seemingly lengthy, is not as laborious as it appear to be. The prompts are relatively long mainly because there are multiple stages we would like to automate: from task generation, to scene configuration generation, object configuration generation, and reward generation. For each of these stages, we only provided 2-4 in-context examples to regulate the response of the LLM. In fact, contemporary work using LLMs for producing policy learning typically uses comparable or more in-context examples (e.g. Voxposer [1] used 10 examples). 

[1] Huang, Wenlong, et al. "Voxposer: Composable 3d value maps for robotic manipulation with language models." arXiv preprint arXiv:2307.05973 (2023).

**Regarding novel tasks**: The in-context prompts are mainly used to regularize the output of the LLM so that its response follows an ideal format. We have multiple seeding strategy to increase the diversity of the tasks generated. Note that, since LLMs have encoded common sense knowledge to reason about object affordances, we can feed the LLM any novel sampled objects that are not in the in-context prompts, and the LLM can reason about its affordance and generate correspondingly meaningful task, and then also other objects needed for the tasks. We believe the diversity of tasks are sufficient to demonstrate that the generalization and reasoning capability of the LLM are helpful in **generating novel tasks**. For example, while the prompt we feed to the LLM only includes tasks involving one single type of articulated object (***oven***), the generated tasks include various semantically novel interactions and tasks. We present several examples below:
- **flushing** the toilet
- **retrieving goldbar from the safe**
- **adjusting time** on clock
- **changing lighting direction of a lamp**
- **heating** soup with **microwave**
- adjusting the **lever** for **operating coffee machines**
- **extending suitcase handles**
- **storing** objects in **fridges**

All of these tasks have **novel semantic meanings** and interaction types that are not present in the given prompt, demonstrating that such task generation process indeed extracts the common sense knowledge embedded in LLMs for automated task generation, which are otherwise not possible if using heuristic based procedural generation pipelines. We hope our explanation on this matter help address your concern.

Reviewer comment (Reproducibility):
The method lacks a formal mathematical formula or algorithmic description.

Author response:
## 4. Formula or algorithm
> The method lacks a formal mathematical formula or algorithmic description.

We respectfully contend that GenBot, as a system paper, primarily demonstrates a new paradigm and its initial realization, rather than delving into specific mathematical details. Given that it employs readily replaceable foundation models for its backend modules, we argue that a formal mathematical formulation is not mandatory in its current form.

However, we definitely agree that **having a more structured algorithmic description**, along with more clear structure of the method description, coule greatly help conveying our method and its major components. Per your suggestion, we have **added a detailed algorithm** describing the whole pipeline in Appendix C. In addition, we have also extensively **revised Section 3.2** into a **structured bullet list** with a more clear logical flow. We hope our updated manuscript is more clear in describing our method!

Reviewer comment (Experiments):
The experiments are insufficient to validate the idea thoroughly.

Author response:
## 5. Request for more thorough experiments
> The experiments are insufficient to validate the idea thoroughly.

Thank your for bringing up this concern! We acknowledge that our initial submission could definitely benefit from more extensive quantitative experiments. We have **significantly expanded the experimental section** in our revised paper, including statistical **evaluation of generated tasks, scenes and skills, as well as more failure analyses**. We explain in details below:

- **Diversity**: In our original submission, we compared with several popular human-designed manipulation task suites in terms of semantic diversity in language space using both Self-BLUE and embedding similarity in Table 1. We now added **additional** diversity comparison also in *image space*, using both pre-trained ViT and CLIP models to compute the embedding similarity of the rendered images of the scenes (see updated Table 1 and associated explanations). We copy the main results below. Compared to other human-designed tasks suites, GenBot achieves a lower image-space embedding similarity measured using both ViT and CLIP. We believe our current comparison in both language and image space, with different metrics and different embedding models, could sufficiently demonstrate the diversity of our generated tasks. 

|                                           | GenBot    | Behavior-100 | RLbench | MetaWorld | Maniskill2 |
|-------------------------------------------|-----------|--------------|---------|-----------|------------|
| Number of Tasks                           | 106       | 100          | 106     | 50        | 20         |
| Task Description (Self-BLEU)            ↓ | **0.284** | 0.299        | 0.317   | 0.322     | 0.674      |
| Task Description (Embedding Similarity) ↓ | **0.165** | 0.210        | 0.200   | 0.263     | 0.194      |
| Scene Image Embedding Similarity (ViT)  ↓ | **0.193** | 0.389        | 0.375   | 0.517     | 0.332      |
| Scene Image Embedding Similarity (CLIP) ↓ | **0.762** | 0.833        | 0.864   | 0.867     | 0.828      |

- **Scene validity**: In order to verify the validity of the generated scenes, we added a round of human evaluation of the generated scenes on whether the generated scene aligns with the language annotations, and if the scene configurations and retrieved objects are correct for the task. (**See updated parts in section 4.2 and 4.3**) We generated 155 tasks and performed human evaluations on them (full list in Appendix D). There are 12 failures due to incorrect scene generation, leading to a **92% validity rate** of our generated scene. (See **scene validity** in section 4.3 for failure case analysis.) 
- **Decomposition and reward function validity**: In order to further verify the validity of the generated task decomposition and reward functions, we added an additional round of human evaluations on the generated training supervisions, by asking a human expert to carefully inspect whether the generated decompositions and reward functions are reasonable and could in principle lead to successful skill learning. We also manually inspected the generated decompositions and reward functions, and found 6 failure cases in the 155 generated tasks (a **96% validity rate**). We revised corresponding paragraphs in section 4.2 and 4.3 with failure analysis, and included **full details of all the tasks and additional statistics in Appendix D**.

Reviewer comment (Experiments):
The low-level RL skills use object states as observations, which seems not able to fully utilize the advantages of diverse visual appearance introduced by task generation.

Author response:
## 6. Questions
> The low-level RL skills use object states as observations, which seems not able to fully utilize the advantages of diverse visual appearance introduced by task generation, could authors provide further explanation about this?

This is a very good question! In fact, this is done intentionally, and is actually one of the key advantages in scaling up demonstration collection in *simulation*. As we stated above, the main goal of our framework is to automate and scale up *demonstration* generation associated with diverse tasks and environments, rather than building an ultimate unified policy conditioned on language and vision input. The latter will be made possible once we have a scalable way for collecting demonstration data. Therefore, in order to generate the demonstration for each task, having access to these low-level state in simulaiton is a clear advantage, which reduces the dimension of observation space in RL training, leading to much more efficient policy learning compared to training vision-based policy. In fact, this type of "cheating" for obtaining demonstration is a common technique used in numerous related works [1][2]. On the other hand, the diversity in visual apperance in the tasks generated, will be used in the downstream distillation, which is to train a unified and *vision-conditioned* policy using the *generated* demonstrations and imitation learning, potentially leading to a multi-task generalist policy that operates using vision and language input. The latter, as we discussed above, has been a long-standing goal for robotics research, and we believe our proposed paradigm and framework for automating demonstration generation takes one important step towards this goal.

[1] Chen, Tao, Jie Xu, and Pulkit Agrawal. "A system for general in-hand object re-orientation." Conference on Robot Learning. PMLR, 2022.
[2] Lin, Xingyu, et al. "Diffskill: Skill abstraction from differentiable physics for deformable object manipulations with tools." arXiv preprint arXiv:2203.17275 (2022)

Reviewer comment (Experiments):
The paper does not provide insights into how the acquired skills can be effectively reused to accomplish new goals when faced with a novel task during testing. Reporting experimental results in novel task settings through skill reusing would further strengthen the paper.

Author response:
> Additionally, could the paper provide insights into how these acquired skills can be effectively reused to accomplish new goals when faced with a novel task during testing? Reporting experimental results in novel task settings through skill reusing would further strengthen the paper;

Thank you for asking this. As we discussed above, the focus of our work is to *generate* demonstrations, rather to study training a unified policy in a lifelong/continual learning setting. Therefore, generating *diverse enough* tasks, and generate *demonstration* given any generated task *efficiently* is our main focus. Skill reusing in case of novel tasks or novel goals is an orthogonal research direction and is beyond the scope of this work. That said, we agree that this is a interesting research direction, and we believe our data generation pipeline could be beneficial to research in related directions.
On a related note, a relevant improvement in the current pipeline could be generating the task in a more structured curriculum, and generate new tasks based on previously similar tasks and re-initialize policy using the existing skills in similar task settings. This could potentially help improve the generation efficiency, and we are happy to explore this in the next step.

Reviewer comment (Theory):
The proposed pipeline seems to be cascaded, where the errors or infeasibility produced at the task generation stage could lead to future difficulties in skill learning. The authors have not discussed any strategies to improve the interplay between task generation and skill acquisition.

Author response:
> The proposed pipeline seems to be cascaded, where the errors or infeasibility produced at the task generation stage could lead to future difficulties in skill learning, have the authors ever considered any strategies to improve the interplay between task generation and skill acquisition?

Thank you for this suggestion! We have added a number of experiments analyzing the validity and success rate of each stage, as we described above. We hope these quantitative number help convince the robustness of our system. Another point we would like to clarify is, in the downstream policy distillation stage, we could incorporate additional feedback using e.g. latest GPT-4V model to help filter out unsuccessful skills, and distill using only successful demonstrations. Another interesting idea is to use feedback from skill learning (success/failure, duration taken, etc.) to iteratively improve the scene configuration and fix potential issue in the scene generation, as you suggested earlier. Such iterative feedback is also used in contemporary work using LLMs for reward generation [1]. We leave such improvement to our future work.

[1] Ma, Yecheng Jason, et al. "Eureka: Human-Level Reward Design via Coding Large Language Models." arXiv preprint arXiv:2310.12931 (2023).


Thank you again for your constructive feedback and suggesting these room for potential improvements! They are really helpful for improving the quality of our submission. We hope our response, clarification and revised manuscript could address your major concerns.

Reviewer comment (Theory):
The assumption that the decomposed shorter-horizon sub-task can be solvable by one of the policy categories within this framework is needed. However, this is not a drawback unique to this research, so it may not be considered a weakness.

Author response:
Thank you for bringing up this! Indeed, LLM-based task decomposition and planning is by itself an active research area, and our system could benefit from using the latest progress in this direction.

We have also added a number of additional evaluations and failure case analysis in section 4.3 and Appendix D, regarding the validity of the generated scenes, the stats of task decompositions and success rate of learned skills after decomposition.

Reviewer comment (Reproducibility):
Are the physical characteristics of objects (e.g., weight) also determined by LLM during generation?

Author response:
In our current pipeline, we use the default rigid contact model and density in the simulation. We only ask the LLM to generate the size. That said, it's definitely possible to query for additional information such as mass and friction coefficient between different properties of surfaces, which are well supported by GPT-4 since the massive text data (e.g. knowledge in wikipedia) it has been trained on encodes such information. We will also add this in our future version. Thanks for this suggestion!

Reviewer comment (Evaluation):
Were there cases where the decomposed shorter-horizon sub-task could not be resolved within one of the policy categories in this framework?

Author response:
We added a round of quantitative analysis on the skill learning success rate, i.e., given a set of generated tasks (with correct training supervisions), if we run the skill learning pipeline for a fixed number of times, how many of those would be successful in learning the skill. The success in learning a skill is determined by a human evaluator watching the video of the learned policy. We test the success rate of our skill learning pipeline on a subset of 50 generated object manipulation tasks, and achieve an average success rate of 0.75, i.e., if we run the skill learning pipeline for 4 times, 3 of the runs will successfully learn the skill. Details on the individual success rate on each of the 50 tasks can be found in **Appendix D.2**.

Reviewer comment (Reproducibility):
What is the overall computational cost involved in the framework?

Author response:
Task generation is simply done by calling GPT-4's APIs, and generating one single task typically can be done within few minutes.
If the task's subgoals can all be solved via planning, typically each task can be solved within 10 minutes. If certain subgoals requires RL to solve, it usually takes around 2-3 hours for each RL-necessary step, and the total duration thus depends on both the number and nature of the subtasks. Taking these into account, a task typically takes 4-5 hours on average to solve. This is done using 8 threads of a CPU running at 2.5Ghz, meaning that *each* node with a 32-core (64-thread) CPU in a typical cluster could run 8 jobs in parallel at the same time. We have added these into Appendix D.1.

Reviewer comment (Evaluation):
I am concerned regarding the diversity of the generated tasks. With tasks proposed by LLMs and only qualitative examples provided, it's challenging for readers to gauge the true diversity of these tasks.

Author response:
Thanks you for bringing this up! 
In our original submission, we compared with several popular human-designed manipulation task suites in terms of semantic diversity in language space using both Self-BLUE and embedding similarity in Table 1. We now added **additional** diversity comparison also in *image space*, using both  ViT (imagenet pre-trained) and CLIP models to compute the embedding similarity of the rendered images of the scenes (see updated Table 1 and associated explanations). We copy the main results here:
|                                           | GenBot    | Behavior-100 | RLbench | MetaWorld | Maniskill2 |
|-------------------------------------------|-----------|--------------|---------|-----------|------------|
| Number of Tasks                           | 106       | 100          | 106     | 50        | 20         |
| Task Description (Self-BLEU)            ↓ | **0.284** | 0.299        | 0.317   | 0.322     | 0.674      |
| Task Description (Embedding Similarity) ↓ | **0.165** | 0.210        | 0.200   | 0.263     | 0.194      |
| Scene Image Embedding Similarity (ViT)  ↓ | **0.193** | 0.389        | 0.375   | 0.517     | 0.332      |
| Scene Image Embedding Similarity (CLIP) ↓ | **0.762** | 0.833        | 0.864   | 0.867     | 0.828      |

Compared to other human-designed tasks suites, GenBot achieves a lower image-space embedding similarity measured using both ViT and CLIP. Regarding the example you gave on "walking forward" and "walking backward", their image space embedding will be very similar, so we think this additional image space comparison would be helpful in evaluating the diversity from another axis. We believe our current comparison in both language and image space, with different metrics and different embedding models, could more convincingly demonstrate the diversity of our generated tasks. 

We also included additional details of a set of 155 generated tasks in Appendix D.1, together with their task name, task description, number of task decompositions, and skill learning success rate.
When developing our pipeline, for implementation simplicity, we considered both table-top manipulation and general manipulation setting without a table. For tasks that GPT-4 deem requiring a table, we sample from a pool of 5 tables with different geometries, textures and heights (see our[ website](https://generativesimulation.github.io/) for detailed visuals). We believe this is already more complex and diverse than existing human-designed table-top manipulation benchmarks (like RLBench and MetaWorld). That said, it's definitely doable to treat table the same way as other objects instead of considering it separately. We leave this unification as our future work.

Reviewer comment (Experiments):
How many semantically distinct tasks are generated? By "semantically distinct," I refer to tasks that are fundamentally different. For instance, "opening a cabinet" and "lifting a bucket" are semantically distinct, whereas "walking forward" and "walking backward" are not.

Author response:
Thanks you for bringing this up! 
In our original submission, we compared with several popular human-designed manipulation task suites in terms of semantic diversity in language space using both Self-BLUE and embedding similarity in Table 1. We now added **additional** diversity comparison also in *image space*, using both  ViT (imagenet pre-trained) and CLIP models to compute the embedding similarity of the rendered images of the scenes (see updated Table 1 and associated explanations). We copy the main results here:
|                                           | GenBot    | Behavior-100 | RLbench | MetaWorld | Maniskill2 |
|-------------------------------------------|-----------|--------------|---------|-----------|------------|
| Number of Tasks                           | 106       | 100          | 106     | 50        | 20         |
| Task Description (Self-BLEU)            ↓ | **0.284** | 0.299        | 0.317   | 0.322     | 0.674      |
| Task Description (Embedding Similarity) ↓ | **0.165** | 0.210        | 0.200   | 0.263     | 0.194      |
| Scene Image Embedding Similarity (ViT)  ↓ | **0.193** | 0.389        | 0.375   | 0.517     | 0.332      |
| Scene Image Embedding Similarity (CLIP) ↓ | **0.762** | 0.833        | 0.864   | 0.867     | 0.828      |

Compared to other human-designed tasks suites, GenBot achieves a lower image-space embedding similarity measured using both ViT and CLIP. Regarding the example you gave on "walking forward" and "walking backward", their image space embedding will be very similar, so we think this additional image space comparison would be helpful in evaluating the diversity from another axis. We believe our current comparison in both language and image space, with different metrics and different embedding models, could more convincingly demonstrate the diversity of our generated tasks. 

We also included additional details of a set of 155 generated tasks in Appendix D.1, together with their task name, task description, number of task decompositions, and skill learning success rate.
When developing our pipeline, for implementation simplicity, we considered both table-top manipulation and general manipulation setting without a table. For tasks that GPT-4 deem requiring a table, we sample from a pool of 5 tables with different geometries, textures and heights (see our[ website](https://generativesimulation.github.io/) for detailed visuals). We believe this is already more complex and diverse than existing human-designed table-top manipulation benchmarks (like RLBench and MetaWorld). That said, it's definitely doable to treat table the same way as other objects instead of considering it separately. We leave this unification as our future work.

Reviewer comment (Experiments):
What is the range of diversity in scene configurations? Upon reviewing the prompts, it seems that certain elements, like a table, have fixed poses and heights. If this is a recurring theme, then scene configuration diversity appears limited.

Author response:
Thanks you for bringing this up! 
In our original submission, we compared with several popular human-designed manipulation task suites in terms of semantic diversity in language space using both Self-BLUE and embedding similarity in Table 1. We now added **additional** diversity comparison also in *image space*, using both  ViT (imagenet pre-trained) and CLIP models to compute the embedding similarity of the rendered images of the scenes (see updated Table 1 and associated explanations). We copy the main results here:
|                                           | GenBot    | Behavior-100 | RLbench | MetaWorld | Maniskill2 |
|-------------------------------------------|-----------|--------------|---------|-----------|------------|
| Number of Tasks                           | 106       | 100          | 106     | 50        | 20         |
| Task Description (Self-BLEU)            ↓ | **0.284** | 0.299        | 0.317   | 0.322     | 0.674      |
| Task Description (Embedding Similarity) ↓ | **0.165** | 0.210        | 0.200   | 0.263     | 0.194      |
| Scene Image Embedding Similarity (ViT)  ↓ | **0.193** | 0.389        | 0.375   | 0.517     | 0.332      |
| Scene Image Embedding Similarity (CLIP) ↓ | **0.762** | 0.833        | 0.864   | 0.867     | 0.828      |

Compared to other human-designed tasks suites, GenBot achieves a lower image-space embedding similarity measured using both ViT and CLIP. Regarding the example you gave on "walking forward" and "walking backward", their image space embedding will be very similar, so we think this additional image space comparison would be helpful in evaluating the diversity from another axis. We believe our current comparison in both language and image space, with different metrics and different embedding models, could more convincingly demonstrate the diversity of our generated tasks. 

We also included additional details of a set of 155 generated tasks in Appendix D.1, together with their task name, task description, number of task decompositions, and skill learning success rate.
When developing our pipeline, for implementation simplicity, we considered both table-top manipulation and general manipulation setting without a table. For tasks that GPT-4 deem requiring a table, we sample from a pool of 5 tables with different geometries, textures and heights (see our[ website](https://generativesimulation.github.io/) for detailed visuals). We believe this is already more complex and diverse than existing human-designed table-top manipulation benchmarks (like RLBench and MetaWorld). That said, it's definitely doable to treat table the same way as other objects instead of considering it separately. We leave this unification as our future work.

Reviewer comment (Experiments):
The construction of tasks in simulation typically requires validation to ensure correct implementation. This involves examining success conditions, initial state distributions, physical parameters, and more. However, the paper lacks a systematic method for this crucial verification, especially given the automation of task creation. Mistakes at any stage could result in flawed tasks.

Author response:
Thank you for your suggestion on this matter! In order to address these, we added the following analysis:
- In order to verify the validity of the generated scenes, we added a round of human evaluation of the generated scenes on whether the generated scene aligns with the language annotations, and if the scene configurations and retrieved objects are correct for the task. (See updated parts in section 4.2 and 4.3) We generated 155 tasks and performed human evaluations on them (full list in Appendix D). There are 12 failures due to incorrect scene generation, leading to a 92% valid rate of our generated scene. (See **scene validity** in section 4.3 for failure case analysis.) 
- In order to further verify the validity of the generated task decomposition and reward functions, we added an additional round of human evaluation on the generated training supervisions, by asking a human expert to carefully inspect whether the generated decompositions and reward functions are reasonable and could in principle lead to successful skill learning. We also manually inspected the generated decompositions and reward functions, and found 6 failure cases in the 155 generated tasks (a 96% validity rate). We revised corresponding paragraphs in section 4.2 and 4.3 with failure analysis, and included full details in Appendix D.3.
- We now also provide a quantitative analysis on the skill learning success rate, i.e., given a set of generated tasks (with correct training supervisions), if we run the skill learning pipeline for multiple times, how many of those would be successful in learning the skill. The success in learning a skill is determined by a human evaluator watching the video of the learned policy. We test the success rate of our skill learning pipeline on a subset of 50 generated object manipulation tasks, and achieve an average success rate of 0.75, i.e., if we run the skill learning pipeline for 4 times, 3 of the runs will successfully learn the skill. Details on the individual success rate on each of the 50 tasks can be found in **Appendix D.2**.

We did not add explicit filtering for simple tasks. To justify this, we would like to elaborate a bit further on the main motivation for this automated pipeline: our proposed paradigm called *generative simulation*, does not intend to generate only hard tasks and serve as a fixed set of benchmarking tasks. Instead, the main goal is to automate the pipeline for generating diverse tasks and their associated demonstration trajectories (i.e. scaling up data collection in simulation), so that they can potentially be used to distill into a multi-task language-conditioned policy later. With this goal, tasks with various levels of difficulties all carry importance to ensure the distribution of data used to train a downstream policy is wide enough, resulting in better generalization. 
In addition, since this is a relatively new research direction, there lacks a well-established and systematic verification process to verify all the components rigorously, and we believe this is also an opportunity for future research in this direction. The main contribution of GenBot is a comprehensive proof-of-concept, and to showcase the feasibility and effectiveness of this novel research direction. In this first attempt, we believe our current newly added and extensive human evaluation is informative in demonstrating its potential, and would hopefully open up numerous research possibilities. That said, one clear improvement in our near-future work is to replace current VLMs with the newly released GPT-4V model, and investigate how much human effort in evaluation can be further reduced with vision feedback in the latest VLM model.

Reviewer comment (Evaluation):
What percentage of the tasks can be successfully solved? How does this compare to the total number of generated tasks?

Author response:
Thank you for your suggestion on this matter! In order to address these, we added the following analysis:
- In order to verify the validity of the generated scenes, we added a round of human evaluation of the generated scenes on whether the generated scene aligns with the language annotations, and if the scene configurations and retrieved objects are correct for the task. (See updated parts in section 4.2 and 4.3) We generated 155 tasks and performed human evaluations on them (full list in Appendix D). There are 12 failures due to incorrect scene generation, leading to a 92% valid rate of our generated scene. (See **scene validity** in section 4.3 for failure case analysis.) 
- In order to further verify the validity of the generated task decomposition and reward functions, we added an additional round of human evaluation on the generated training supervisions, by asking a human expert to carefully inspect whether the generated decompositions and reward functions are reasonable and could in principle lead to successful skill learning. We also manually inspected the generated decompositions and reward functions, and found 6 failure cases in the 155 generated tasks (a 96% validity rate). We revised corresponding paragraphs in section 4.2 and 4.3 with failure analysis, and included full details in Appendix D.3.
- We now also provide a quantitative analysis on the skill learning success rate, i.e., given a set of generated tasks (with correct training supervisions), if we run the skill learning pipeline for multiple times, how many of those would be successful in learning the skill. The success in learning a skill is determined by a human evaluator watching the video of the learned policy. We test the success rate of our skill learning pipeline on a subset of 50 generated object manipulation tasks, and achieve an average success rate of 0.75, i.e., if we run the skill learning pipeline for 4 times, 3 of the runs will successfully learn the skill. Details on the individual success rate on each of the 50 tasks can be found in **Appendix D.2**.

We did not add explicit filtering for simple tasks. To justify this, we would like to elaborate a bit further on the main motivation for this automated pipeline: our proposed paradigm called *generative simulation*, does not intend to generate only hard tasks and serve as a fixed set of benchmarking tasks. Instead, the main goal is to automate the pipeline for generating diverse tasks and their associated demonstration trajectories (i.e. scaling up data collection in simulation), so that they can potentially be used to distill into a multi-task language-conditioned policy later. With this goal, tasks with various levels of difficulties all carry importance to ensure the distribution of data used to train a downstream policy is wide enough, resulting in better generalization. 
In addition, since this is a relatively new research direction, there lacks a well-established and systematic verification process to verify all the components rigorously, and we believe this is also an opportunity for future research in this direction. The main contribution of GenBot is a comprehensive proof-of-concept, and to showcase the feasibility and effectiveness of this novel research direction. In this first attempt, we believe our current newly added and extensive human evaluation is informative in demonstrating its potential, and would hopefully open up numerous research possibilities. That said, one clear improvement in our near-future work is to replace current VLMs with the newly released GPT-4V model, and investigate how much human effort in evaluation can be further reduced with vision feedback in the latest VLM model.

Reviewer comment (Experiments):
Are trivial tasks, such as picking up a block when given the grasp action primitive, filtered out?

Author response:
Thank you for your suggestion on this matter! In order to address these, we added the following analysis:
- In order to verify the validity of the generated scenes, we added a round of human evaluation of the generated scenes on whether the generated scene aligns with the language annotations, and if the scene configurations and retrieved objects are correct for the task. (See updated parts in section 4.2 and 4.3) We generated 155 tasks and performed human evaluations on them (full list in Appendix D). There are 12 failures due to incorrect scene generation, leading to a 92% valid rate of our generated scene. (See **scene validity** in section 4.3 for failure case analysis.) 
- In order to further verify the validity of the generated task decomposition and reward functions, we added an additional round of human evaluation on the generated training supervisions, by asking a human expert to carefully inspect whether the generated decompositions and reward functions are reasonable and could in principle lead to successful skill learning. We also manually inspected the generated decompositions and reward functions, and found 6 failure cases in the 155 generated tasks (a 96% validity rate). We revised corresponding paragraphs in section 4.2 and 4.3 with failure analysis, and included full details in Appendix D.3.
- We now also provide a quantitative analysis on the skill learning success rate, i.e., given a set of generated tasks (with correct training supervisions), if we run the skill learning pipeline for multiple times, how many of those would be successful in learning the skill. The success in learning a skill is determined by a human evaluator watching the video of the learned policy. We test the success rate of our skill learning pipeline on a subset of 50 generated object manipulation tasks, and achieve an average success rate of 0.75, i.e., if we run the skill learning pipeline for 4 times, 3 of the runs will successfully learn the skill. Details on the individual success rate on each of the 50 tasks can be found in **Appendix D.2**.

We did not add explicit filtering for simple tasks. To justify this, we would like to elaborate a bit further on the main motivation for this automated pipeline: our proposed paradigm called *generative simulation*, does not intend to generate only hard tasks and serve as a fixed set of benchmarking tasks. Instead, the main goal is to automate the pipeline for generating diverse tasks and their associated demonstration trajectories (i.e. scaling up data collection in simulation), so that they can potentially be used to distill into a multi-task language-conditioned policy later. With this goal, tasks with various levels of difficulties all carry importance to ensure the distribution of data used to train a downstream policy is wide enough, resulting in better generalization. 
In addition, since this is a relatively new research direction, there lacks a well-established and systematic verification process to verify all the components rigorously, and we believe this is also an opportunity for future research in this direction. The main contribution of GenBot is a comprehensive proof-of-concept, and to showcase the feasibility and effectiveness of this novel research direction. In this first attempt, we believe our current newly added and extensive human evaluation is informative in demonstrating its potential, and would hopefully open up numerous research possibilities. That said, one clear improvement in our near-future work is to replace current VLMs with the newly released GPT-4V model, and investigate how much human effort in evaluation can be further reduced with vision feedback in the latest VLM model.

Reviewer comment (Evaluation):
The majority of the results are qualitative, which lacks depth for readers. Additionally, the paper's comparison of task diversity to other benchmarks based solely on task descriptions is less than persuasive.

Author response:
We acknowledge that due to the complexity of the system, our initial submission could benefit from more extensive quantitative experiments. We have added numerous experiments as described **above** in terms of both diversity measure, statistical evaluation of **generated tasks, scenes and skills, as well as failure analysis** (see 1 and 2 above for details). We hope the revised version is more persuasive now.

Reviewer comment (Reproducibility):
As of now, the paper hasn't released its code. Furthermore, the underlying simulation framework, "Genesis," remains private.

Author response:
Thank you for suggesting this! Genesis is still not publicly available yet and its various APIs are subject to change, therefore only open to a group of researchers participating in internal testing. The tentative time for releasing Genesis is next Feburary according to the development team.

We very much agree open-sourcing the code would aid the reproducibility of our method and benefit the research community. We convert GenBot to a re-implementation of GenBot using pybullet as the main backend instead, and have **open-sourced this version of code** **[here](https://github.com/GenerativeSimulation/GenBot/tree/main/gpt_4/prompts)** in a anonymous repo. We will release the full pipeline as soon as Genesis is released publicly. We hope this could help address your concern!

Reviewer comment (Experiments):
How does the system handle incompatibilities between the robot and the object? For instance, if the robot is a dog and the object is a cabinet, what would the proposed task be?

Author response:
We apologize for not explaining this clearly in the initial version. For tasks involving legged robot, we prompt the LLM with only example-based seeding, and populate objects only when the LLM says additional objects is required for the task. We have updated section 3.1 to make this clear.

Reviewer comment (Experiments):
In scenarios involving non-articulated objects, if the generated tasks aren't specifically tied to the sampled object, how does the sampled object influence the task?

Author response:
In such case, the object will be populated into the scene as part of the scene components, but won't necessarily be interacted with in task learning, if the sub-goal or reward function do not involve it. On the other hand, in example-based seeding, the examples are just used to hint the LLM for generating different tasks, and the objects in examples will be not used in creating the actual scene.

Reviewer comment (Reproducibility):
Could you provide a more detailed explanation of the MidJourney + Zero123 pipeline?

Author response:
We added more details on this in Appendix E. See **5. Object Assets** above for more detailed explanation.

Reviewer comment (Reproducibility):
What is the precise output format from the LLMs? How is this output imported into a simulator to construct a scene?

Author response:
The LLM will generate a `yaml` file describing the scene configuration. See **Scene Generation Prompt** in Appendix F for examples. This yaml file will then be parsed by yaml parser to load the specified objects and generate the scene in the simulator.

Reviewer comment (Experiments):
How does the system manage situations where the scene results in an unsolvable task? For example, if the task is to open a cabinet but the cabinet is positioned out of the robot arm's reach.

Author response:
We specifically added support for robot manipulators with **mobile base** so that such reachability issue won't be a typical failure cause. There are also other potential failure modes on which we added an analysis in section 4.3 (also see detailed response above). Note that our pipeline's generated tasks and skills achieves a validity rate over 90% and skill learning success rate over 75%.

Reviewer comment (Experiments):
Is the initial state of the scene fixed, or is it sampled from a distribution?

Author response:
Currently the initial state is given by the LLM, and we resolve any detected collision. We set the temperature properly (see Appendix A) so that the response from GPT-4 is stochastic while being reasonable. Therefore, each time the generated task and scene configuration will be different. We could also use this initial configuration as an initialization and add random disturbance sampled from a gaussian distribution, which could further enhance the diversity of the scene. In the current version we are not yet doing this.

Reviewer comment (Reproducibility):
How does the system generate physical parameters other than size, such as friction?

Author response:
In our current pipeline, we use the default rigid contact model in the simulation and do not change the friction. We only ask the LLM to generate the size. That said, it's definitely possible to query for additional information such as mass and friction coefficient between different properties of surfaces, which are well supported by GPT-4 since the massive text data (e.g. knowledge in wikipedia) it has been trained on encodes such information. In order to prove this, we conducted a quick test using GPT-4 to return the mass and friction coefficient of several objects when they are in contact with a **wooden table surface**, and the response is below:

| Object       | Estimated Mass (kg) | Estimated Friction Coefficient (μ) |
|--------------|---------------------|------------------------------------|
| Microwave    | 15                  | 0.5                                |
| Water Bottle | 1                   | 0.4                                |
| Lamp         | 3                   | 0.5                                |
| Laptop       | 2                   | 0.3                                |
| Clock        | 1                   | 0.4                                |

We believe these value are reasonable enough for serving as a initial guess followed by random perturbations to produce diversity.
We will also add this in our future version. Thanks for this suggestion!

Reviewer comment (Evaluation):
How reliable is the reward generated by the LLM? Are there instances where it may not align with the intended goal?

Author response:
Please see our above response in evaluating the reward functions and skills learned, as well as added detailed in Section 4.3.

Reviewer comment (Evaluation):
How does the system define the success conditions for a task?

Author response:
Currently, we do not have explicit success condition generation, and we evaluate the final system and learned skills with human evaluation. That said, there's indeed concurrent work trying to generate success condition using LLMs (1), similar to reward generation. We will try to incorporate this in our pipeline. We would like to re-iterate that the main contribution of our work is proposing a potentially promising paradigm, and presenting a concrete and comprehensive implementation of it to prove the validity and effectiveness of this new direction. There are definitely plenty of room for potential improvement, employing the rapid developing research directions such as multi-modal foundation models and 3D generative models. Since GPT-4V's API has just been released, one clear direction is to incorporate its vision feedback to both improve robustness in generating scenes and tasks, as well as evaluating if learned skills match the intended goal.

[1] Ha, Huy, Pete Florence, and Shuran Song. "Scaling up and distilling down: Language-guided robot skill acquisition." arXiv preprint arXiv:2307.14535 (2023).

Reviewer comment (Experiments):
Experiments need to be more thorough for each of the components (task diversity, scene validity, training supervision validity, etc).

Author response:
Indeed, we acknowledge that the experiments in our initial submission are not rigorous enough, potentially affecting the persuasiveness of our paper. We are grateful for your valuable suggestion on this matter! We have significantly expanded the experimental section in our revised paper with numerous new experiments and quantitative analyses, including quantitative evaluation on **task diversity** (4.1), **scene validity** (4.2), **training supervision validity** (4.3), **skill learning** (4.4) and **system** (4.5). We discuss them in details below.

Reviewer comment (Presentation):
The text in sections 3.1 and 3.2 could perhaps be shorter; it is a bit difficult to follow with a lot of text including details interspersed throughout.

Author response:
Thank you for your suggestion! Section 3.1 mainly explains how the tasks are proposed, and 3.2 mainly talks about how we generate a corresponding scene based on the proposed task. We acknowledge that our initial presentation might have appeared complex, potentially leading to a perception of excessive complexity and too many parts involved. However, the pipeline is actually simple without too many stages interacting with each other: for generating the scene, we first obtain the names of objects (assets) needed, then either retrieve or generate them (with additional verification for improving the validity of the scene), then putting them into the scene with plausible configurations (poses, configuration of links and joints, etc). Each of these stages is necessary for obtaining a semantically meaningful scene that could lead to successful skill learning.

We made the following changes per your advice to make these sections more clear and easier to follow:

- We have extensively revised Section 3.1 and 3.2, prioritizing the most critical details in the pipeline, while eliminating redundant explanations and unimportant details. This helped shorten the sections by half a page.
- We moved less-important details, such as details of asset retrieval using sentence-bert, using VLMs (Bard and BLIP-2) for verifying the assets, why we use 2 models to cross-validate, and details of the text-to-3D pipeline into the appendix, since these models are not the main contributions of our work and multiple models at play could confuse readers. (In the later part of this response, we also explain why these modules are used and how they can be replaced to be more unified and robust in the future work.)
- We re-structured Section 3.2 into a structured bullet list: we now first talk about the necessary components needed for generating the scene in the first paragraph, and then elaborate details of each of these componenets in the following structured list. To make it more clear, we also restructured the components so that they now match the 4 stages depicted in Figure 2 (B), and obtaining asset queries and their meshes are now grouped together into the first bullet point.
- We also added a algorithmic pipeline in Appendix C to better summarize the whole process.

We hope that the revised version of our paper presents a more structured and focused narrative, and will facilitate a better understanding of our method.

Reviewer comment (Presentation):
Consider an additional graphic for section 3.2, or at least a structured/bullet list, to clarify the many moving parts and how they interact at different stages.

Author response:
Thank you for your suggestion! Section 3.1 mainly explains how the tasks are proposed, and 3.2 mainly talks about how we generate a corresponding scene based on the proposed task. We acknowledge that our initial presentation might have appeared complex, potentially leading to a perception of excessive complexity and too many parts involved. However, the pipeline is actually simple without too many stages interacting with each other: for generating the scene, we first obtain the names of objects (assets) needed, then either retrieve or generate them (with additional verification for improving the validity of the scene), then putting them into the scene with plausible configurations (poses, configuration of links and joints, etc). Each of these stages is necessary for obtaining a semantically meaningful scene that could lead to successful skill learning.

We made the following changes per your advice to make these sections more clear and easier to follow:

- We have extensively revised Section 3.1 and 3.2, prioritizing the most critical details in the pipeline, while eliminating redundant explanations and unimportant details. This helped shorten the sections by half a page.
- We moved less-important details, such as details of asset retrieval using sentence-bert, using VLMs (Bard and BLIP-2) for verifying the assets, why we use 2 models to cross-validate, and details of the text-to-3D pipeline into the appendix, since these models are not the main contributions of our work and multiple models at play could confuse readers. (In the later part of this response, we also explain why these modules are used and how they can be replaced to be more unified and robust in the future work.)
- We re-structured Section 3.2 into a structured bullet list: we now first talk about the necessary components needed for generating the scene in the first paragraph, and then elaborate details of each of these componenets in the following structured list. To make it more clear, we also restructured the components so that they now match the 4 stages depicted in Figure 2 (B), and obtaining asset queries and their meshes are now grouped together into the first bullet point.
- We also added a algorithmic pipeline in Appendix C to better summarize the whole process.

We hope that the revised version of our paper presents a more structured and focused narrative, and will facilitate a better understanding of our method.

Reviewer comment (Novelty):
There is some additional work on learning diverse skills in simulated environments (in some cases in addition to real-world) that should be cited:  
- Jiang et al, 2022, VIMA: General Robot Manipulation with Multimodal Prompts  
- Majumdar et al, 2023, Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?  
- Bousmalis et al, 2023, RoboCat: A Self-Improving Foundation Agent for Robotic Manipulation

Author response:
Thank you for bringing up these relevant works! We have updated the Related Work section with additional citations to the 3 papers you suggested.

Reviewer comment (Experiments):
Given the significant complexity proposed, the experiments should be more thorough and quantitative to do justice to the complexity of the method.

Author response:
Indeed, we acknowledge that the experiments in our initial submission are not rigorous enough, potentially affecting the persuasiveness of our paper. We are grateful for your valuable suggestion on this matter! We have significantly expanded the experimental section in our revised paper with numerous new experiments and quantitative analyses, including quantitative evaluation on **task diversity** (4.1), **scene validity** (4.2), **training supervision validity** (4.3), **skill learning** (4.4) and **system** (4.5). We discuss them in details below.

Reviewer comment (Evaluation):
**Task diversity**: Measuring task diversity using just the language descriptions may be prone to biases (e.g., task suites may describe tasks differently, with different levels of verbosity). Providing the same measures in state space (e.g., diversity of robot joint motions required to solve the tasks) or image space (e.g., final image showing goal configurations for each task) would be more convincing.

Author response:
Thank you for your suggestion. Since measuring diversity in robot joint space could be largely affected by the range of joint motions and the size of the workspace (our generated tasks consider room-size workspace, which is naturally at advantage compared to other table-top manipulation settings), and there lacks any established and systematic methods for measuring task diversity this way, we decided to add additional results measuring diversity in **image space** following your suggestion. We revised section 4.2 and 4.3, and included additional diversity in image space, using both ViT (imagenet pre-trained) and CLIP models to compute the embedding similarity of the rendered images of the scenes (see updated Table 1 and associated explanations). We copy the main results here:
|                                           | GenBot    | Behavior-100 | RLbench | MetaWorld | Maniskill2 |
|-------------------------------------------|-----------|--------------|---------|-----------|------------|
| Number of Tasks                           | 106       | 100          | 106     | 50        | 20         |
| Task Description (Self-BLEU)            ↓ | **0.284** | 0.299        | 0.317   | 0.322     | 0.674      |
| Task Description (Embedding Similarity) ↓ | **0.165** | 0.210        | 0.200   | 0.263     | 0.194      |
| Scene Image Embedding Similarity (ViT)  ↓ | **0.193** | 0.389        | 0.375   | 0.517     | 0.332      |
| Scene Image Embedding Similarity (CLIP) ↓ | **0.762** | 0.833        | 0.864   | 0.867     | 0.828      |


Compared to other human-designed tasks suites, GenBot achieves a lower image-space embedding similarity measured using both ViT and CLIP, in addition to the previously shown better diversity in language space. We hope the updated results is more convincing now.


In addition, we also provide the list of generated tasks, including the task name and task descriptions, as qualitative evaluations of the task semantics diversity in **Appendix D.1**. In case you missed it in the first round of review, we also provided visualizations of the generated tasks on our project [**website**](https://generativesimulation.github.io/) for better quantitative understanding of the scene and robot motion diversity.

Reviewer comment (Experiments):
**Scene validity**: While Figure 4 shows the BLIP-2 scores for asset retrieval (and ablates some of the verification stages), it is not clear (i) how much the method relies on retrieval versus generation (i.e., when retrieval fails), (ii) how viable the generated assets are versus retrieval, and (iii) how important the different moving parts are beyond the specific verification stages (i.e., how important is it to have both Bard and BLIP-2? Why Sentence-Bert?).

Author response:
**Before answering your questions in detail, we would like to further clarify a bit the motivation** behind having both retrieval and generation for obtaining the object assets. This paper's primary contribution, from our perspective, is the introduction of a generative simulation paradigm, merging generative models with physics-based simulation to automate the whole pipeline for scalable and automated robot learning. Our focus therefore lies in establishing a comprehensive and adaptable framework, prioritizing its foundational development over specific backend models for the initial implementation. Considering the rapid advancements in LLMs and 2D/3D generative models, with SOTA models being updated on a weekly basis, our framework is designed to be able to upgrade seamlessly by using the latest progress in these orthogonal research areas; while GenBot, serving as the first comprehensive proof-of-concept, showcases the feasibility and effectiveness of this novel research direction.

While human-designed database for 3D assets is growing rapidly, with massive-scale database like Objaverse-XL with millions of objects, they are mostly designed by human artists and would require massive human effort to continue scaling up. On the other hand, automated text-to-3D generative pipelines utilize massive 2D image resources available online, coupled with inductive biases such as view consistency, holds a better potential in further scaling up and automating the process for obtaining 3D assets. While at the time of submission, retrieval from existing databases usually presents higher-quality meshes and faster speed, we believe text-to-3D pipeline holds a greater potential in the future, in terms of both asset fidelity and generation speed. Therefore, **we intentionally designed our framework to support both of these two pipelines to verify both of them could work**. As a result, in the first implementation and design of GenBot, we tried to unify both pipelines and prioritize retrieval over generation: the initial design we had is resorting to the generative pipeline whenever retrieval fail to pass the verification stage. However, in practice we found that for rigid-body manipulation settings, objects retrieved from Objaverse already work very well and the generative pipeline is rarely needed. Therefore, in practice, we now only use the text-to-3D pipeline for soft-body manipulation tasks, in order to obtain a more consistent and controllable target shape for the soft-body under manipulation. Another reason is that soft-body manipulation's goal shape doesn't require detailed texture, which is also one limitation of generative models at the time of submission. We have revised section 3.2 to make this more clear. We also included some visual results showing the quality of the generated assets in Table 5 in Appendix E.

We would also like to point to some latest work in 3D asset generation (one-2-3-45 [1], instant3d [2], etc.) after our initial submission, which dramatically improved the generation speed and results. The models in our current pipeline can be easily swapped with these latest progresses, which we will incorporate in future development.

[1] https://one-2-3-45.github.io/

[2] https://instant-3d.github.io/

**Regarding using both Bard and BLIP-2**: The main reason for using both is that at the time of submission, GPT-4V is not available yet and these 2 models are among the most advanced VLM models available, while each of them still presents certain limitations. BLIP-2 is trained on a relatively small dataset and doesn't generalize well to data distribution present in Objaverse. Bard is a more general model, but it doesn't provide official APIs and we used a 3rd party [interface](
https://github.com/dsdanielpark/Bard-API) to access it, making its generation temperature beyond our control, hence leading to somewhat inconsistent responses. Also, the vision capability of Bard is still below a satisfactory level at the submission time. We empirically observe that using both model to cross-validate would make the verification process more conservative and hence more robust.

Now that GPT-4V just released its API, we will replace all these modules with GPT4-V in our  future development so that we can remove the messy dependencies on various difference back-end VLMs, which could further improve the performance and robustness of our system.

Reviewer comment (Evaluation):
**Training Supervision Validity**: This would be more convincing with any quantitative results, even something like the average number of decompositions per proposed task; the average duration to solve each full task versus sub-goals; performance if solving the full task directly via planning (if possible), etc.

Author response:
Thanks for raising this up! We added an additional round of human **evaluations on the generated training supervisions**, by asking a human expert to carefully inspect whether the generated decompositions and reward functions are reasonable and could in principle lead to successful skill learning. We found 6 failure cases in the 155 generated tasks (a **96% validity rate**). We revised corresponding paragraphs in section 4.2 and 4.3, and included full details in Appendix D, including average number of decompositions (Figure 6 in Appendix D). The errors can be categorized into 1) referring to undefined variables, e.g., referring to “standard time” in task “Correct Clock Time” yet it is undefined; 2) reward does not encode the intended behavior. This includes incorrect semantic understanding of articulated object state, e.g., the task is to fold the chair, yet the generated reward actually encourages unfolding the chair due to misunderstanding of the mapping between joint angle values and object state. We also find it hard to generate correct rewards for continuous motions such as “moving robotic hand back-and-forth”, or “knock the door”. However, our reward generation component can be further improved by incorporating environment feedback into the system following very recent work focused on reward generation using LLMs [1], which we leave as future work.

Solving the long-horizon tasks without sub-goals is not practically feasiable, since solving the task requires reaching the intermediate goal specified by the sub-goal state and reward functions. Also, planning alone is not sufficient in solving many tasks, as we explained in section 3.3: reinforcement learning better suits tasks that are contact rich and involving continuous interaction with other scene components, e.g., legged locomotion, or when the required actions cannot be simply parameterized by discrete end-effector poses, e.g., turning the knob of an oven. Regarding duration for solving the task: if the task's subgoals can all be solved via planning, typically each task can be solved within 10 minutes. If certain subgoals requires RL to solve, it usually takes around 2-3 hours for each RL-necessary step, and the total duration thus depends on both the number and nature of the subtasks. Taking these into account, a task typically takes 4-5 hours on average. This is done using 8 threads of a CPU running at 2.5Ghz, meaning that *each single* node with a 32-core (64-thread) CPU in a typical cluster could run 8 jobs in parallel at the same time. We have added these into Appendix D.1.

[1] Ma, Yecheng Jason, et al. "Eureka: Human-Level Reward Design via Coding Large Language Models." arXiv preprint arXiv:2310.12931 (2023).

Reviewer comment (Experiments):
**Skill Learning Performance**: The quantitative results show improvement over an RL-only baseline, but it would be more helpful to show this over many more than 4 tasks; and also report the relative performance of all three methods (i.e., separating trajectory optimization and planning over action primitives). Action primitives look pretty high-level: grasping, approaching, and releasing a target object. How often is this route selected? And how much of the performance is due to working with an easier planning problem in a much higher-level action space rather than RL?

Author response:
Thank you for your suggestion. We have replaced the original Table 2 with the new Figure 5 to show quantitative numbers on **12 rigid-body manipulation tasks (instead of 4)**. For each task, we run each method with 4 different seeds, and report the mean and std of the task return (we manually verified that the reward functions generated by GPT-4 are correct for the evaluated tasks). Due to the randomness in the skill learning process (sampling is used in the motion planning-based action primitive, and RL inherently has randomness during exploration and training), **we also provide quantitative analysis on the skill learning success rate**, i.e., given a generated task (with correct training supervisions), if we run the skill learning pipeline for multiple times, how many of those would be successful in learning the skill. The success in learning a skill is determined by a human evaluator watching the video of the learned policy. 

We find that allowing learning algorithm selection is beneficial for achieving higher performance for completing the tasks. When the option of motion planning based action primitive is removed and only RL is used, the skill learning completely fails for most tasks. We test the success rate of our skill learning pipeline on a subset of **50 generated object manipulation tasks**, and achieve an **average success rate of 0.75**, i.e., if we run the skill learning pipeline for 4 times, 3 of the runs will successfully learn the skill. Details on the individual success rate on each of the 50 tasks can be found in **Appendix D.2**. Note that since gradient-baed trajectory optimization is used solely for soft-body manipulation, we do not include comparison with it here. This is a conclusion derived from a series of prior work studying soft-body manipulation setting [1][2][3], all of which suggest fined-grained shape matching task involving soft-body would be better solved with gradient-based optimization technique, and RL or planning would usually fail in solving them. This selection choice is injected to GPT-4's response by including this information in the prompt.

[1] Huang, Zhiao, et al. "Plasticinelab: A soft-body manipulation benchmark with differentiable physics." arXiv preprint arXiv:2104.03311 (2021).

[2] Lin, Xingyu, et al. "Diffskill: Skill abstraction from differentiable physics for deformable object manipulations with tools." arXiv preprint arXiv:2203.17275 (2022).

[3] Xian, Zhou, et al. "Fluidlab: A differentiable environment for benchmarking complex fluid manipulation." arXiv preprint arXiv:2303.02346 (2023).

Reviewer comment (Evaluation):
**System**: The final system performance needs to be a quantitative analysis. As it stands, there is no clear sense of how well the overall method works, in terms of how many different tasks it can solve, to what degree, and the nature of those tasks (e.g., what objects, what behaviour/affordance, etc.).

Author response:
In case you missed it earlier, we have included a number of different tasks proposed and solved using GenBot on our [**website**](https://generativesimulation.github.io/). We have also added details of generated tasks, their statistics, and analysis on failure cases in Appendix D.1, D.2 and D.3. Shortly speaking, our system reached **92% scene validity rate**, **96% training supervision validity rate**, **75% skill learning success rate**, and meanwhile presenting **higher diversity** than popular human-designed task sets.

We hope our response could address your major concerns. We would like to thank you again for the critical and helpful feedbacks and suggestions. They are really helpful in improving the quality of our work!