Paper: GenSim: Generating Robotic Simulation Tasks via Large Language Models (ICLR-2024)

Reviewer comment (Experiments):
Most of the current tasks are within the domain of top-down pick and place, and the generalization of policies within such a domain is relatively easy. It could be more impressive to see more dexterous manipulation tasks and policy generation over those.

Author response:
While the domain of top-down pick-and-place, especially short-horizon tasks, are *seemingly* not challenging to most roboticists, we note that the  **task-level policy generalization** explored in GenSim, such as long-horizon tasks and compositional tasks, remains under-explored and non-trivial. For example, "build-a-car" is a nontrivial task that requires semantic understanding of natural language instructions and more than 10 steps of precise pick-and-place and stacking. We are actively experimenting with extending our pipeline to handle more challenging tasks such as dexterous, 6-dof, contact-rich tasks.

Reviewer comment (Evaluation):
For the failure cases of task generation, perhaps providing more examples will draw a better boundary on the limitation of the current approach, or for each model.

Author response:
Thanks for asking this.  Overall, we acknowledge that even with the strongest model GPT-4 in the current approach, there are still gaps in achieving generation of infinite *high-quality* tasks without any human efforts, due to **the bottleneck of grounding and the hallucination** problems in LLMs at its current state. We have added detailed error messages to Appendix D.4. The GPT-4 model sometimes hallucinates concepts such as “boundary” and “ascending size” without implementing these in code. For weaker models such as Code-LLama and GPT-3.5, the gap from GPT-4 is in generating creative and complex tasks, and also in correctly implementing them in code.

Reviewer comment (Reproducibility):
In Sec. 3.1, for exploration task generation, which LLM is used? Is it fine-tuned or GPT models without fine-tuning? Also for the generation of 120 tasks, are the newly generated tasks used for fine-tuning as well or just as prompt examples? Please provide more explanations.

Author response:
**GPT-4 without finetuning** is used in the exploration task generation to generate the task library. This is because the base GPT-4 model is the current strongest LLM model (See Figure 6 right) and has the desired creativity and reasoning ability to generate new tasks in exploratory task generation stage. In addition to being retrieved in the prompts for GPT-4, these 120 tasks are also used for finetuning the open-source model such as Code LLama and closed-source model such as GPT 3.5. However, as shown in Figure 6 right, even fine-tuning the GPT3.5 or codellama with 120 tasks, the success rate cannot outperform GPT4. Therefore, we choose GPT-4 as the base model to generate gensim tasks.

Reviewer comment (Evaluation):
For Fig. 6, it seems GPT-4 without fine-tuning outperforms all other fine-tuned models, so is it true that using fine-tuned smaller models is just for consideration of inference cost?

Author response:
Lower costs, because of fewer prompt tokens and cheaper models, and better efficiency, because of more structured and domain-specific output, are important motivations for finetuning LLMs. Moreover, we show that scaling up simulation tasks can also work well with open-source LLM models, where we have access to the model itself and potentially can be used for many other purposes. This finetuning process also enables self-instruct improvements as the task library is expanding.

Reviewer comment (Experiments):
As mentioned in Sec. 1 the task library is initialized with only 10 tasks, for fine-tuning the LLMs to gain context of robotic simulation tasks, is the overfitting to these 10 tasks a problem? If so, how is it handled?

Author response:
In finetuning experiments, we used **all the 100 tasks** from the generated task library from GPT-4 to finetune LLMs. Indeed, during the LoRA finetuning process, we have observed some overfitting problems. We expect further scaling up training tasks to conduct self-improvement will improve this issue.

Reviewer comment (Experiments):
Figure 3 mentions that human inspection of a generated task is required. Was feedback collected for each of the generated tasks? How many tasks were manually annotated in total? In addition, what is the granularity of feedback provided by the human?

Author response:
We note that human inspection is *not required* for each task and the effort of annotation only involves watching rendered videos (without the need for checking code).  First, around 90% of erroneous tasks can be filtered out via automatic metrics (e.g., execution errors). Second, for the rest of the tasks, humans only need to check whether the simulated tasks are correct or not by checking whether the rendered videos correspond to the natural language description of the tasks.  In our experiments, LLMs generate around *200* tasks in total, and humans (i.e. the authors) checked and filtered around 150 tasks and kept 100 tasks.  Our new experiment below also shows that the human filtering rate is around 50%.

During the response period, we assess human effort (time and pass rates) quantitatively by re-generating 30 tasks for human feedback. In Table 3 (attached below, also in Appendix Section D.6), we find that the required human efforts are around 10 seconds (for CS students) on average for each task. Note again that this is 10% auto-filtered tasks and the amounts of total inspectations are much smaller than all generated tasks. For future work, we are also exploring whether non-CS humans can check the correctness so that we can scale up the process of collecting human feedback. 

| Task       | User 1 Time | User 1 Pass | User 2 Time |User 2  Pass | User 3 Time |User 3 Pass |
|------------|------------|------------|------------|------------|------------|------------|
| Average    | 12.83 | 0.8 | 5.11  | 0.63 | 17.4 | 0.41 |

You can also get a sense of how much effort is needed for verification by trying out our Gradio demo (https://huggingface.co/spaces/Gen-Sim/Gen-Sim).

Reviewer comment (Experiments):
The experiments presented in Table 1 use a different set of tasks than the real tasks used to evaluate CLIPort. In particular, the tasks do not involve sweeping, or folding of objects. The variety of asset is also smaller than what is shown in CLIPort where scissors or chess pieces are manipulated. A similar observation is made regarding the synthetic experiments, e.g., the held-out simulation tasks used for evaluation mostly use blocks and omit more complex objects like ropes. Additional explanations on how these tasks were selected are needed to convince the readers that the method indeed generalizes to a wide variety of tasks. Experiments presented in Fig 6 (left) evaluate models on the same tasks as CLIPort but it is not clear if the models were pre-trained on all generated tasks in this experiment (see W3).

Author response:
We have conducted additional experiments to verify that our framework can also extended to more varied tasks and assets as suggested. In simulation, we evaluate new tasks such as **separating-piles**.
|            | single task+0     | Single task + 2    | Single task +4    |
|------------|------------|------------|------------|
| Average    | 29.7 | 40.1| 49.3 |

As shown in the above table, the experiment results are also consistent. The GPT-generated tasks demonstrate efficacy across a more diverse array of objects. 
 In the real world, we have also evaluated novel tasks with assets such as fruits and spoons.
|            |   organize-kits  | sort-garbage  | place-bread   |
|------------|------------|------------|------------|
| Average    | 60.0 | 40.0 | 30.0 |

We have updated Figure 8 and Table 1 in the main paper to reflect the new experiments, we showed that training and evaluating on more diverse tasks such as **spoons and piles** are also possible, while the main bottleneck is the hardware. Given our time budget, we showed that the policies pre-trained from 70 GenSim tasks can outperform policies pre-trained from 10 CLIPort tasks and training from scratch. Due to the limited time and hardware limitations (only suction cup), we chose these tasks (see [website](gen-sim.github.io) for visualizations), and we believe our pipeline and pretrained weights can be useful for more tasks.

There are two reasons behind our task selection: task-level generalization and sim-to-real transfer.

1. We highlight that we did not focus our evaluation on diverse assets such as rope, because we focus more on *task-level generalization*. We consider “build-a-car” to also be a challenging task that involves. We note that in *Table 2*, we have already shown performance on tasks on “move-piles-along-line” and “manipulating-two-ropes”. We believe that when scaling is improved, we will see more interesting generalizations to both diverse tasks and scenes due to the robustness of ``common-sense`` representations. 

2. We point out that our experiment focuses on pretraining on large simulation data and transferring it to the real world. For CLIPort, it only focused on sim-to-sim and real-to-real for demonstration purposes. In fact, in Table 1, similar zero-shot sim-to-real baselines will have 0 success rates due to the perception gaps. However, we are interested in evaluating novel-task generalization in simulation and conducting sim-to-real by using policies bootstrapped from the simulation. We also use a different hardware than the CLIPort (we use xARM with a suction cup), so dexterous tasks with very complex geometry in the real world can be difficult. 

> Experiments presented in Fig 6 left) evaluate models on the same tasks as CLIPort but it is not clear if the models were pre-trained on all generated tasks in this experiment (see W3.)

Thanks for asking. We would like to know clarifications on the Figures. It looks like that you might be referring to *Figure 7* since *Figure 6* does not include CLIPort experiments. We do not pre-train on the evaluation tasks or pre-train on any tasks in this few-shot generalization experiment. Moreover, the GPT-generated training tasks are selected based on task code embeddings produced by LLM model “text-embedding-ada-002”. Please feel free to clarify the question and let us know if this needs a follow-up.

Reviewer comment (Experiments):
The “few-shot policy generalization to related tasks” experiment needs additional description to understand the meaning of “single-original, single+2, single+4 etc”. Were the models pre-trained on original CLIPort tasks, original CLIPort tasks + 2 generated tasks etc.? Or are the models pre-trained on all CLIPort tasks + all the ones that were generated automatically? It is important to clarify this experiment as the current version lets the reader think that only a small subset of the generated tasks were used to pre-train the policy, and the question of how these were selected arises.

Author response:
Thanks for your suggestion.  We have added additional descriptions for Figure 7 and the few-shot policy generalization section in the main paper. The x-axis denotes single CLIPort task + N GPT tasks where N=0,2,4 and multiple CLIPort tasks + N GPT tasks where N=0,4,8. We use the GPT embedding to pick related tasks from the evaluation tasks. The goal of the experiment is to see if the additional related tasks and data generated by GPT-4 can be used to improve the training performance. We clarify that the models are not pre-trained on any tasks. This experiment does not have large-scale pre-training, we train few-shots on CLIPort (single/multiple)+N tasks from GenSim and evaluate the CLIPort tasks.

Reviewer comment (Evaluation):
Page 6 mentions “When only pretraining on the 10 tasks in CLIPort, the policy does not generalize well on the GPT4 tasks.” In Fig 7 (right) the model pre-trained on tasks generated with LLMs is better than the one trained on CLIPort tasks, but the difference is 15-20% of relative improvements, so the claim should therefore be mitigated. It is also suggested to justify how the set of testing tasks (given in Appendix A.1) was selected. See also W2.

Author response:
We acknowledge that the claim on the performance of pre-training on CLIPort tasks should be mitigated and we have removed that sentence. However, we would like to point to the additional results, which state that pre-trained CLIPort tasks indeed only have limited zero-shot generalization in our simulation experiments (only 1.6% success rate among 4 GPT generated tasks). We also note that the right subfigure in Figure 7 is few-shot joint training instead of zero-shot generalization. The tasks in Appendix A.1 are used for evaluating LLM task generation across different LLMs (from GPT-3 finetuned to GPT-4). The set of testing tasks is not particularly selected.

Reviewer comment (Evaluation):
Details of the metrics used in Fig 6 would be appreciated. In particular, how is “code reasoning capability” measured in “runtime-verified” and is the “task completed” an average over several environment resets?

Author response:
There are two levels of mistakes that LLMs make. The metric mostly measures low-level mistakes. Our metric has an incremental structure from syntax to runtime to successfully generate demonstrations where failing the former metrics implies failing the latter metrics. “Runtime verified” measures if the code can be runnable without any runtime problems such as out of bound, “Task completed” is computed as 1 if we have 3 trials managing to generate demonstrations over 5 trials (sometimes there could be simulation issues). Therefore, they test whether the LLM can successfully write the code to generate demonstrations. Our additional human experiments also measure that the human pass rate is around 0.5. We also note that evaluating LLM judiciously is still an open research question.

Reviewer comment (Experiments):
Simplistic tasks. The experimentation is limited to table-top pick-and-place task. Furthermore, the use of a suction gripper is a weakness -- this simplifies the already simple task, and limits generalizability to other kinds of tasks. Unclear whether the proposed methodology can be adapted for more complex tasks, and how well it would do. Experiments on more complex tasks would be more convincing.

Author response:
While the domain of top-down pick-and-place, especially short-horizon tasks, are *seemingly* not challenging to most roboticists, we note that the  **task-level policy generalization** explored in GenSim, such as long-horizon tasks and compositional tasks, remains under-explored and non-trivial. For example, "build-a-car" is a nontrivial task that requires semantic understanding of natural language instructions and more than 10 steps of precise pick-and-place and stacking. Such task-level generation cannot be achieved by previous methods without LLMs, as future work we plan to extend our work to handle more challenging tasks such as dexterous, 6-dof, contact-rich tasks.

Reviewer comment (Experiments):
A number of approaches exist in the literature for data augmentation using foundation models (e.g. ROSIE, CACTI, etc.). How do such approaches compare to the proposed methodology when it comes to generalization to unseen tasks? Arguably, some of these are simpler to implement and use.

Author response:
Thanks for bringing up these interesting works; we will include these references in the related work section. From our understanding, ROSIE and CACTI focus more on **scene-level generalization** on a single task, utilizing image diffusions for data augmentation. However, in our work, we focus on **task-level** generalization with LLM for generating more tasks. As these works are orthogonal to our approach, it would be interesting to explore a combination of both to achieve generalization across unseen tasks in diverse scenes.

Reviewer comment (Reproducibility):
What is done in the cases where the output of the LLM has a syntactical error and cannot be run?

Author response:
One of the properties of coding simulation tasks is that we can just execute the robot simulation and check the results. Therefore, if the output of LLM is non-executable, we will do **rejection sampling** and move to the next trial of LLM generation. We have added a new Appendix Section D.5 on the feedback in GenSim. Feel free to check that section out.

Reviewer comment (Reproducibility):
How much prompt engineering went into this?

Author response:
During the LLM pipeline development process, we went through prompt engineering, retrievals and reflections, and finetuning and used techniques such as chain-of-thoughts and few-shot prompting. The final version of the prompt is *not* very complicated and contains two subtasks of generating the task descriptions and the task code (Figure 3). Figure 6 Left shows the necessity of this subtask separation in prompt chains to give LLM enough “time” to think. The prompts are attached in Appendix Section D.2. One intuition is that creating simulation task descriptions and implementing the code for the tasks require different language capabilities. For creating the task, we would like to generate tasks as creative and interesting as possible (with high temperature for decoding) while being physically feasible, so it tests the language reasoning capabilities of LLM. For implementing the task code in simulation, it requires rigorous coding execution (with 0 temperature for decoding) and needs to be thoughtful in the details of syntax and goal generation. In summary, the two subtask pipeline is important and this can avoid complicated prompt engineering.