Paper: Make a Donut: Language-Guided Hierarchical EMD-Space Planning for Zero-shot Deformable Object Manipulation (ICLR-2024)

Reviewer comment (Evaluation):
**W1 - Technical insights are unclear for the deformable and the general robotics manipulation community.** On pre-existing tasks, the paper attempts to show that LLM can generate geometrical representations of intermediate subgoals that guide action planning. It seems that the take-home message for the paper is that we can build a planning framework with LLM to do these pre-existing tasks. However, the results and analysis provided by the authors **do not convince me that existing frameworks such as PASTA[1] cannot accomplish the proposed tasks**. Specifically, the result section lacks discussion, visualization, and analysis of (1) why baseline approaches like PASTA did not perform well, (2) which part of the pipeline is responsible for the failure (e.g., perception, planning, control), and (3) how do the proposed LLM planning techniques in this paper overcome these challenges. Another suggestion I would like to give is to come up with a task that previous approaches really struggled to complete. Currently, the capability demonstrated in this paper, in both simulation and the real world, does not convince me that the proposed approach outperforms prior works.

Author response:
We appreciate your critical evaluation of the technical insights presented in our paper. The tasks of creating dough shapes such as donuts and baguettes are indeed not pre-existing benchmarks within deformable object manipulation; thus, they offer a novel challenge that extends beyond the capabilities of established frameworks like PASTA.

The key distinction of our approach lies in its zero-shot learning ability, which enables it to adapt to novel tasks without task-specific fine-tuning or training. PASTA, while effective within its demonstrated scope, requires a dataset to train on in order to sample feasible intermediate states during planning and is thus inherently limited in its ability to generalize to new shapes, e.g., donut. All their modules like VAE, cost predictor, etc., are tailored to their collected training data. This data-driven dependency hinders its application to the more complex tasks our framework successfully tackles. We also want to point out that PASTA needs to specify the desired number of stages to execute before the planning, and we already helped them out by specifying 2 stages for making a donut, 2 stages for making a baguette and 3 for making two pancakes, but their results are still worse than ours. You can find some visualizations of PASTA in our updated A.5 (sorry to put that in the appendix due to space limit).

We have now included a detailed analysis in the results section that elucidates the limitations of PASTA and similar baseline approaches. This includes a discussion on their reliance on extensive datasets and the inability to adapt to tasks involving complex manipulations such as squeezing, which our method can handle adeptly. Please see the Results paragraph of section 4.1 in our updated manuscript. We have augmented our manuscript with analyses pinpointing the failure introduced by PASTA’s inherent plan learning algorithm. This breakdown helps clarify the innovative aspects of our LLM-based planning technique.

Our LLM hierarchical planning framework's strength lies in its generative capability, which allows it to conceptualize intermediate subgoals that are not only geometrically feasible but also physically plausible. This is a significant leap over the sampled-based methods, which may not provide a feasible path or require extensive data for complex shapes. The Large Language Model (LLM) plays a crucial role in our framework by charting a high-level planning path, which serves as a guide for the subsequent execution by the low-level Earth Mover's Distance (EMD) space planning algorithm. This hierarchical structure is pivotal; the LLM alone cannot translate its generated plans into the raw actions required for robotic manipulation. Conversely, without the strategic direction provided by the LLM, the EMD space planning lacks a coherent objective, struggling to discern what end states are physically plausible for the robot to achieve. This interdependence is crucial for successful task completion and is evidenced by the comparative results in Figure 5(a), Section 4.3, where the EMD space planning alone is shown to be inadequate without the LLM's high-level input.

To ensure the message is clear and convincing, we have enriched the paper with the requested discussions, visualizations, and comparative analyses. This additional content will reinforce our argument that our LLM-based planning framework is indeed a significant advancement over previous methods like PASTA, particularly in its zero-shot generalization capabilities and task adaptability.

Reviewer comment (Experiments):
**W2 - Unconvincing real robot experiments.** In the real robot experiments showcased in the supplementary video, the approach struggles to complete the desired shapes. For instance, at timestamp 01:51 of the video, the robot struggles to split the dough into halves. Plus, a human operator, in almost all tasks, needs to hold the rubber cushion below the dough to make physical interactions stable. In addition, textual and pictorial details are missing in both the video and the paper. It would be helpful for the authors to provide **the desired final shape**, **the intermediate shapes generated by LLM**, and **the achieved shapes after real robot executions**. It would also be helpful for the authors to discuss how the incorporation of LLM makes a difference, and which part of the pipeline is responsible for failure cases.

Author response:
We are grateful for your insights on the real robot experiment demonstrations. We acknowledge the instances where the robot does not perfectly execute the splitting of the dough, as highlighted at the 01:51 timestamp.

The human operator's presence was solely to address the stickiness of the dough, which adhered to the rubber cushion, a non-interactive element of the environment. This was a measure taken to ensure the continuity of the experiment and in no way contributed to the task's execution, as the operator did not interact with the dough or the robot arm during the manipulation process.

We have since made strides to improve our setup, including the modifications to secure the cushion in place. These improvements will be reflected in additional experiments and visualizations, which we are committed to providing in a few days and will update them in the response.

To ensure that our paper presents a comprehensive and transparent view of our research, we are committed to including a thorough depiction of the entire process from initial planning to execution. This will encompass detailed descriptions and visual representations of the intended final shapes, the intermediate subgoals as envisioned by the LLM, and the actual outcomes achieved by the robotic execution. This addition will be featured in the appendix section of our submission, where we have already provided visualizations for the simulated environments in Section A.5 of the updated manuscript.

We acknowledge that there is an inherent challenge in attaining flawless execution in deformable object manipulation, a difficulty that extends to human practitioners due to the innate variability in material behavior and environmental factors. The target shapes proposed by the LLM, while geometrically precise, may not always be perfectly replicated in a physical setting. Despite this, our method demonstrates a significant capability—it consistently selects the appropriate tools and approaches to create shapes that adhere to the correct topology and general structure required for the task. This is a notable improvement over prior methodologies such as PASTA, which may not reliably select the optimal tools for each step of the process.

Dear reviewer, we are pleased to inform you that we have made major updates to our real-world experimental setup by removing the rubber cushion, where there is no human intervention now. We invite you to view the improved execution in the supplementary videos, particularly from the 1:00 timestamp onwards, which showcases the real robot execution following our planning algorithm.
Additionally, we have included image visualizations of the robot's real-world execution in section A.9 of the appendix. 
We hope these updates address the concerns previously highlighted, and appreciate any feedback or further discussions.

Reviewer comment (Theory):
**W3 - The paper lacks a discussion on the limitations of optimal transport.** The paper incorporates many techniques that hinge on optimal transport measures being faithful. However, many works such as [2] and [3] have discussed that optimal transport is limited to heavy geometrical characterization of shape differences, and lacks the physical prior to capture the "physical distance" to transform one dough shape into another. Please see Figure 4. in [2] as well as the discussion of topology preservation in [3] when directly using optimal transport gradients to transform the shape, which is what the authors are doing in this paper. It would be helpful for the authors to provide discussions on how the proposed LLM approach could face the optimal-transport-related challenges in the limitation section.

Author response:
Thank you for bringing up the critical discourse on the limitations of optimal transport (OT) in capturing the "physical distance" necessary for transforming one dough shape into another. Indeed, the works you cited correctly identify the challenges associated with using OT in isolation, particularly concerning topological preservation and its pure geometrical approach to shape transformation.

Our framework acknowledges these limitations and proposes a novel integration of LLM with OT to address them. The LLM's role is pivotal as it injects a higher-level understanding of the task at hand, including the physical and practical considerations necessary for the manipulation of deformable objects. This is not a mere direct application of OT gradients; rather, the LLM informs the OT process by setting subgoals that are physically meaningful, which are then fine-tuned by the subsequent EMD space planning algorithm.

Specifically, the LLM plans for subgoals that are not only geometrically reachable but also physically plausible, considering the material properties and manipulation capabilities. This is evidenced by our method's ability to create a hole in a doughnut, a task that necessitates a nuanced understanding of both the dough's physical properties and the desired end shape's topology.

Moreover, in our ablation study presented in Figure 5(a), we demonstrate that direct application of OT is insufficient for complex, multi-step tasks and that the integration of LLM's high-level planning is critical. The LLM planning phase ensures that subsequent OT steps are guided towards physically realizable subgoals, thus mitigating the risks identified by the works you referenced.

We hope this explanation clarifies our approach and its distinction from the direct use of OT gradients. The integration of LLM with OT is, we believe, a significant step forward in deformable object manipulation.

Reviewer comment (Experiments):
**W4 - Missing visualization of the next reachable point cloud.** The authors propose to use optimal transport gradients to geometrically transform the shape. As I mentioned in W3, this transformation does not preserve the topological structures of the initial shape and could lead to planning errors. The system therefore could struggle to generalize. It would be helpful for the authors to visualize the next reachable point cloud obtained from warping the initial shape, and how faithful can the low-level differentiable optimizer recover that shape. It would also be helpful to discuss failure cases due to incorrect next-reachable point clouds in the limitation section.

Author response:
In Section A.6 of our revised document, we have incorporated detailed visualizations of the reachable point flow to clearly illustrate the manipulation process. The visualizations demonstrate key stages, such as the intentional alteration of the dough's topology to create a hole in the middle of a donut. In this context, the intrinsic property of optimal transport (OT) that does not necessarily preserve topological structures is leveraged to our advantage. By manipulating the dough in a manner that does not preserve its initial topology, we can achieve the desired end shape—a task that is particularly challenging for traditional methods that strictly adhere to topological constraints. 

The next reachable point cloud is also an integral part of our method's iterative correction process. Although our tools may not execute the point cloud to perfection, the LLM-generated subgoals guide the gradient flow towards the target. The methodology allows for self-correction with each observation, ensuring that each step brings us closer to the intended outcome. We recognize that there are instances where our method may produce donuts with imperfections due to deviations in the early stages of the task, as outlined in section A.7.2 of the appendix. This limitation highlights the potential for future enhancements. With the progression of LLM models capable of incorporating precise 3D feedback and the employment of highly dexterous robotic hands, we anticipate that the execution of tasks can be refined to achieve even closer alignment with the intended outcomes.

Reviewer comment (Novelty):
**Minor Remarks - *Incorrect characterization* of an existing work.** > Existing works on dough-like volumetric deformable objects majorly rely on a learned dynamic model of the underlying particles Zhu et al. (2022); Arriola-Rios et al. (2020); Yin et al. (2021); **Huang et al. (2021)**. Huang et al. (2021) does not rely on a data-driven or "learned" dynamics model. The paper proposes to leverage a built-in differentiable physics engine for trajectory optimization and is thus misrepresented in this citation. I propose that you change this to a more appropriate citation.

Author response:
We appreciate your diligence in ensuring the accurate portrayal of existing work. We have moved PlasticineLab to the discussion of related works on differentiable physics.

We hope that our responses have adequately addressed your concerns and provided a clearer understanding of our framework's contributions to the field. We are committed to improvement and would greatly appreciate any further feedback you may have.

Reviewer comment (Novelty):
The solution proposed in this work is heavily tailored towards the deformable object manipulation setting. It also requires the point-cloud representation and a differentiable physics simulator. This limits the range of applications of the method.

Author response:
We acknowledge your observation regarding the specificity of our framework towards deformable object manipulation. This focus is intentional, as deformable objects present a unique set of challenges distinct from rigid body manipulation. The paper aims to advance the field within this defined scope, contributing a versatile planning framework tailored to the complexities of deformable materials. The utilization of point-cloud representations and a differentiable physics simulator aligns with contemporary methodologies in the field[1][2][3], ensuring that our approach is both relevant and comparable to existing research.

[1]Li, S., Huang, Z., Chen, T., Du, T., Su, H., Tenenbaum, J. B., & Gan, C. (2023). DexDeform: Dexterous Deformable Object Manipulation with Human Demonstrations and Differentiable Physics. arXiv preprint arXiv:2304.03223.

[2]Li, S., Huang, Z., Du, T., Su, H., Tenenbaum, J. B., & Gan, C. (2022). Contact points discovery for soft-body manipulations with differentiable physics. arXiv preprint arXiv:2205.02835.

[3] Xingyu Lin, Carl Qi, Yunchu Zhang, Zhiao Huang, Katerina Fragkiadaki, Yunzhu Li, Chuang Gan, and David Held. Planning with spatial-temporal abstraction from point clouds for deformable object manipulation. In Conference on Robot Learning (CoRL), 2022c.

Reviewer comment (Theory):
The LLM's plan is open-loop. In this case, the next stage assumes the first stage is feasible. When the planner fails to complete a stage, the next stage starts at a bad initial state.

Author response:
The planning framework we propose is inherently flexible and does not rely on the success of previous stages, as each subgoal generated by the LLM is independent. The online EMD space planning algorithm is robust to variations in initial states and is capable of directing the manipulation towards the desired subgoal at each stage though not perfect. We also provide visualizations on intermediate subgoals and the next reachable point cloud at each timestep to reach the subgoal, in the updated A.5, A.6. We acknowledge the complexity of implementing a closed-loop system for LLM that involves real-time perception and reconstruction. While we recognize this as an important direction for future research, our current work is focused on the planning aspect within the constraints of available technology.

Reviewer comment (Experiments):
The input to the LLM is limited to the language description of the dough. If the LLM can get a more accurate/grounded idea of the dough configuration, it might be able to provide higher-quality plans.

Author response:
Your point about the LLM's input is well-taken. Indeed, a more nuanced understanding of the dough's configuration could potentially enhance plan quality. Our framework is built to be compatible with advancements in perception technologies. We aim to address these perceptual challenges in future work, as our current focus is to establish a robust planning strategy within the confines of present-day perception capabilities.

Reviewer comment (Reproducibility):
Can you provide more details on the baseline methods used in the simulated experiments? For example, what is the observation space for the BC and SAC agents?

Author response:
In our experiments, all baseline methods, including BC and SAC agents, utilize the point cloud of the dough as their observation space, plus the current translation and rotation of the tools. This standardization ensures a fair and consistent comparison across different methods. We use a PointNet-like backbone to encode the input point cloud with embedding dim set to 128. For SAC, the learning rates for actor and critic are set to 3e-4, with the reward decaying ratio $\gamma$ set to 0.99.

Reviewer comment (Theory):
Maybe I missed something, but why is the numerator needed in equation 3? It appears to me that neither $\mathbf{p}_i$ nor $\mathbf{p}'_i$ is a function of $\mathbf{x}$.

Author response:
Thank you for your question regarding the numerator in Equation 3. Upon reviewing the equation, we have realized that there was an omission in the manuscript. We apologize for this oversight. The equation indeed requires a summation symbol to accurately represent the aggregation of the terms across all indices \( i \).
The corrected form of Equation 3 should be:
$ \mathbf{x}^* = \arg\max_\mathbf{x} \sum_i \frac{\|\mathbf{p}_i' - \mathbf{p}_i\|_1} { \mathrm{sdf}_x(\mathbf{p}_i) + \delta}.$
It depends on the numerator if we write out all the terms and find a common denominator.

We hope this clarifies your concerns and we appreciate any further feedback you may have.

Reviewer comment (Experiments):
It appears that the LLM does not undergo any interactions or corrections after generating the point clouds for each stage.

Author response:
We appreciate your insight on the role of LLM in our framework. The LLM indeed provides a high-level planning blueprint. Our EMD space planning algorithm is designed to be responsive and adaptive, using the LLM's output as a guide while continuously refining the approach based on real-time observations. This dynamic interplay ensures that potential discrepancies between the subgoals and the physical execution are adjusted for at each step. You can also find more visualizations in our updated A.5 and A.6.

Reviewer comment (Experiments):
The method described is effective for complex tasks with long-term planning, but it can only generate simple shapes using Python code.

Author response:
Your observation regarding shape complexity is astute. While our current system does indeed focus on simpler shapes, the framework's core contribution lies in its general approach to integrating LLM with spatial planning. The complexity of generated shapes is acknowledged as a limitation and an avenue for future research. Our focus remains on establishing a robust planning methodology, which can later be enhanced to tackle more complex shapes as text-to-3D technologies advance.

Reviewer comment (Evaluation):
Error recovery seems incomplete, and there still appears to be a risk of getting stuck after resetting the tool. There is no comprehensive mechanism for early termination.

Author response:
Our framework incorporates a continuous correction mechanism within the EMD space planning algorithm, which actively seeks to reconcile deviations from the intended subgoal after each action, which can be further validated by the provided visualizations in A.5 and A.6. We recognize the importance of an early termination condition and have introduced a termination criterion within our algorithm, as detailed in L23 of Algorithm 1 of the revised manuscript, to prevent potential deadlocks and ensure efficient task completion.

Reviewer comment (Experiments):
Is the human effort required to develop these prompts more costly than define traditional task schema?

Author response:
The effort required to develop prompt templates is minimal when compared to the traditional task schema definition. The use of LLM for planning is grounded in its ability to interpret and act on the context provided by the template. This is a one-time effort, after which users can interact with the system using simple, intuitive prompts. This significantly reduces the cognitive load and technical expertise required from the user.

Reviewer comment (Experiments):
The paper includes the hyperparameters used for dough simulation in A2, but it lacks analysis or discussion on the selection process and their impact on the results. Given that the physical properties of the dough can vary, how should this issue be addressed to make it more useful for different setups?

Author response:
The selection of hyperparameters for dough simulation is consistent with established benchmarks, specifically the PASTA framework, to ensure a fair comparison. We also conduct an experiment to demonstrate the robustness of our approach across a reasonable range of material properties. 
|            | Donut      | Baguette   | TwoPancakes |
|:---------- |:----------:|:----------:|:-----------:|
| Yield stress=200, Friction=1.5 | 0.365/75%  | 0.523/75%  | 0.864/70%   |
| Yield stress=150, Friction=1.0 | 0.332/70%  | 0.512/75%  | 0.842/65%   |
| Yield stress=150, Friction=0.5 | 0.346/75%  | 0.501/75%  | 0.858/65%   |
We agree that extending this robustness to a wider array of materials presents an interesting challenge for future work, which would involve teaching the LLM to interpret and adapt to varying physical dynamics.

Reviewer comment (Evaluation):
You extract the generated Python code from each stage to generate intermediate target point clouds. How to ensure that this generated point clouds are plausible for the real world setting, and how to align them?

Author response:
Our experimental evidence supports the feasibility of LLM-generated intermediate goals for common bakery items like donuts, baguettes, and pancakes. While there may be limits to the complexity of shapes the current tools can produce, we believe these are reasonable constraints given the scope of this paper. Aligning these goals with the real world is achieved by standardizing the initial frame of reference between the LLM coordinate system and the physical environment.

Reviewer comment (Reproducibility):
How to use parallel grippers to manipulate rolling pins and other tools?

Author response:
This objective can be attained by installing a grasp-compatible structure on the tools or by adjusting the gripper's opening distance to securely accommodate the tool dimensions. Our configuration is designed to ensure the gripper makes optimal contact with the tools, whether it be clasping the midpoint of a rolling pin or the apex of a cutter. These adjustments are rudimentary and not the primary focus of our research.