Original: D A XB ENCH : B ENCHMARKING D EFORMABLE O BJECT M ANIPULATION WITH D IFFERENTIABLE P HYSICS
Revised: D A XB ENCH : B ENCHMARKING D EFORMABLE O BJECT M ANIPULATION WITH D IFFERENTIABLE P HYSICS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deformable Object Manipulation (DOM) is of significant importance to both daily and industrial applications.
Revised: Deformable object manipulation (DOM) is a long-standing challenge in robotics and has attracted significant interest recently.

Original: Recent successes in differentiable physics simulators allow learning algorithms to train a policy with analytic gradients through environment dynamics, which significantly facilitates the development of DOM algorithms. However, existing DOM benchmarks are either single-object-based or non-differentiable.
Revised: This paper presents DaXBench, a differentiable simulation framework for DOM.

Original: This leaves the questions of 1) how a task-specific algorithm performs on other tasks and 2) how a differentiable-physics-based algorithm compares with the non-differentiable ones in general. In this work, we present DaXBench, a differentiable DOM benchmark with  a wide object and task coverage. DaXBench includes 9 challenging high-fidelity simulated tasks, covering rope, cloth, and liquid manipulation with various difficulty levels.
Revised: In contrast with existing work that often focuses on a specific type of deformable objects, DaXBench supports fluid, rope, cloth ...; it provides a general-purpose benchmark to evaluate widely different DOM methods, including planning, imitation learning and reinforcement learning. DaXBench combines recent advances in deformable object simulation with JAX, a high-performance computational framework.

Original: To better understand the performance of general algorithms on different DOM tasks, we conduct comprehensive experiments over representative DOM methods, ranging from planning to imitation learning and reinforcement learning. In addition, we provide careful empirical studies of existing decision-making algorithms based on differentiable physics, and discuss their limitations, as well as potential future directions.
Revised: All DOM tasks in DaXBench are wrapped with OpenAI Gym APIs, for fast and easy integration with DOM algorithms. We hope that DaXBench provides to the research community a comprehensive, standardized benchmark and a valuable tool that supports the development and evaluation of new DOM methods.

Original: The code and video can be found in the supplementary files.
Revised: The code and video are available online ∗ .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deformable object manipulation (DOM) is critical to various applications,ranging from household scenarios (Maitin-Shepard et al., 2010; Miller et al., 2011;
Revised: Deformable object manipulation (DOM) is a crucial area of research with a wide range of applications, from household scenarios (Maitin-Shepard et al., 2010; Miller et al., 2011;

Original: Ma et al., 2022) to industrial applications (Miller et al., 2012; Zhu et al., 2022).
Revised: Ma et al., 2022) to industrial settings (Miller et al., 2012; Zhu et al., 2022).

Original: Built upon deformable object simulators, DOM benchmarks have provided a platform for algorithm development and prototyping (Lin et al., 2020; Ma et al., 2022).
Revised: To aid in algorithm development and prototyping, several DOM benchmarks (Lin et al., 2021; Huang et al., 2021) have been developed using deformable object simulators.

Original: However, given the extremely high-dimensional statespace and action space on a DOM task, learning a generalizable and effective policy remains challenging.
Revised: However, due to the high dimensionality of state and action spaces in DOM tasks, creating a generalizable and effective policy is still a significant challenge.

Original: In contrast to standard non-differentiable simulators, differentiable physics simulation constructs physics rules as differentiable computational graphs, which allows direct optimization of a control policy with analytical environment gradients (Freeman et al., 2021;
Revised: Differentiable physics is a promising direction for developing control policies for deformable objects. It implements physical laws as differentiable computational graphs (Freeman et al., 2021;

Original: Hu et al., 2020). Because of their particle-based or mesh-based representations, deformable objects can be naturally connected with differentiable simulations. By directly propagating thegradients from optimization objectives, e.g. , cumulative rewards, to a learning policy through environment dynamics, we can use the groundtruth simulation gradientsto reason about the dynamics and improve sample efficiency during policy learning (Chen et al., 2022;
Revised: Hu et al., 2020), enabling the optimization of control policies with analytical gradients and therefore improving sample efficiency. Recent studies have shown that differentiable physics-based DOM methods can benefit greatly from this approach (Chen et al., 2022;

Original: Hu et al., 2020; Huang et al., 2021; Xu et al., 2022; Heiden et al., 2021), which is favorable in the context of DOM.
Revised: Hu et al., 2020; Huang et al., 2021; Xu et al., 2022; Heiden et al., 2021).

Original: However, most ofthe existing differentiable DOM benchmarks focus on a single task, and provide no evidence to the generalization ability of the resulting task-specific algorithms.
Revised: Therefore, to facilitate fair comparison and further advancement of DOM techniques, it is important to develop a general-purpose simulation platform that accommodates these methods.

Original: In this work, we present DaXBench, a differentiable and comprehensive high-performance benchmark forDOM. DaXBench relies on a Deformable object simulator, DaX, which connects the recent advances of deformable object simulation algorithms (Xu et al., 2022; Chen et al., 2022) to the high-performance computational framework, JAX (Bradbury et al., 2018). In particular, powered by JAX, DaX enjoys the auto-differentiation and parallelization to multiple devices.
Revised: We present DaXBench, a differentiable simulation framework for deformable object manipulation (DOM).

Original: From the task setup perspective, in contrast to single-task-based benchmarks, DaXBench covers a wide range of object types, including rope, cloth, liquid, and elasto-plastic object manipulations.
Revised: In contrast to current single-task benchmarks, DaXBench encompasses a diverse range of object types, including rope, cloth, liquid, and elasto-plastic materials.

Original: Considering object types, a wide range of tasks with varyingdifficulty levels  and well-defined reward functions are presented, such as liquid pouring, cloth folding, and rope wiping.
Revised: The platform includes tasks with varying levels of difficulty and well-defined reward functions, such as liquid pouring, cloth folding, and rope wiping.

Original: Specifically, we design tasks with high-level macro actions and low-level control-based action spaces to fully examine the performance of  different algorithms.
Revised: These tasks are designed to support high-level macro actions and lowlevel controls, enabling comprehensive evaluation of DOM algorithms with different action spaces.

Original: All task environments are wrapped with OpenAI Gym APIs (Brockman et al., 2016) and Python interfaces, such that DaXBench can be seamlessly connected with a DOM algorithm for fast and easy development.

Original: In addition, we benchmark  competitive DOM methods covering all algorithmic paradigms, namely sampling-based planning, Imitation Learning (IL), and Reinforcement Learning (RL).
Revised: We benchmark eight competitive DOM methods across different algorithmic paradigms, including sampling-based planning, Reinforcement Learning (RL), and Imitation Learning (IL).

Original: Specifically, for planning methods, we consider CrossEntropyMethod-MPC (CEM-MPC)
Revised: For planning methods, we consider CrossEntropyMethod-MPC (CEM-MPC)

Original: (Richards, 2005), differentiable MPC (Hu et al., 2020), and a combination of the two, Diff-CEM-MPC.
Revised: (Richards, 2005), differentiable MPC (Hu et al., 2020), and a combination of the two.

Original: For RL domains, we considerstandard Proximal PolicyOptimization (PPO)
Revised: For RL domains, we consider Proximal Policy

Original:  (Schulman et al., 2017) with nondifferentiable dynamics,  Short-Horizon Actor-Critic (SHAC)
Revised: Optimization (PPO) (Schulman et al., 2017) with non-differentiable dynamics, and Short-Horizon Actor-Critic (SHAC)

Original: (Xu et al., 2022), and Analytic Policy Gradients (APG) (Freeman et al., 2021).
Revised: (Xu et al., 2022) and Analytic Policy Gradients (APG) (Freeman et al., 2021) with analytical gradients.

Original: For IL, Transporter networks (Seita et al., 2020) with nondifferentiable dynamics and Imitation Learning via Differentiable physics (ILD) (Chen et al., 2022) are compared.
Revised: For IL, Transporter networks (Seita et al., 2021) with non-differentiable dynamics and Imitation Learning via Differentiable physics (ILD) (Chen et al., 2022) are compared.

Original: In our experiments, by comparing the algorithms with and without analytic gradientsside-by-side on each task, we provide deeper understanding of the benefits and challenges to analytic policy optimization in DOM.
Revised: Our experiments compare algorithms with and without analytic gradients on each task, providing insights into the benefits and challenges of differentiable-physics-based DOM methods.

Original: We discuss the limitations of existing methods and potential future directions with DaXBench.
Revised: DaXBench implements a deformable object simulator, DaX, which combines recent advancements in deformable object simulation algorithms (Xu et al., 2022; Chen et al., 2022) with the highperformance computational framework JAX (Bradbury et al., 2018). This integration allows for efficient auto-differentiation and parallelization across multiple accelerators, such as multiple GPUs. All task environments are wrapped with OpenAI Gym APIs (Brockman et al., 2016), enabling seamless integration with DOM algorithms for fast and easy development. By providing a comprehensive and standardized simulation framework, we aim to facilitate algorithm development and advance the state of the art in DOM.

Original: In addition, we demonstrate that the dynamics model  in DaXBenchis sufficiently realistic that allows direct sim-to-real transfer to a real robot on rope manipulation tasks.
Revised: Moreover, our results demonstrate that the dynamics model used in DaXBench enables direct sim-to-real transfer to a real robot for rope manipulation tasks, demonstrating the potential applicability of our simulation platform to real-world problems.

Original:  R ELATED W ORKS 2. D EFORMABLE O BJECT S IMULATORS
Revised:  R ELATED W ORKS 2. D EFORMABLE O BJECT S IMULATORS

Original: The progress in Deformable Object Manipulation (DOM)  partially stems from therecent emergence of DOM simulators in the past years.
Revised: Recent advancements in Deformable Object Manipulation (DOM) have been partially attributed to the emergence of DOM simulators in recent years.

Original: SoftGym (Lin et al., 2020) is the first DOM benchmark that models all liquid, fabric, and elastoplastic objects and introduces a wide range of standardized DOM tasks.
Revised: SoftGym (Lin et al., 2021) is the first DOM benchmark that models all liquid, fabric, and elastoplastic objects and introduces a wide range of standardized DOM tasks.

Original: ThreeDWorld
Revised: ThreeDWorld

Original: (Gan et al., 2020) allows agents to physically interact with various objects in rich 3D environments. However, they are non-differentiable and thus do not support methods based on differentiable physics.
Revised: (Gan et al., 2020) enables agents to physically interact with various objects in rich 3D environments, but it is non-differentiable and does not support methods based on differentiable physics.

Original: On the other hand, several differentiable DOM simulators emerge, such as ChainQueen (Hu et al., 2018b), Diff-PDE (Holl et al., 2020), PlasticineLab (Hu et al., 2020), DiSECt (Heiden et al., 2021), and DiffSim (Qiao et al., 2020).
Revised: On the other hand, several differentiable DOM simulators emerge, such as ChainQueen (Hu et al., 2019), Diff-PDE (Holl et al., 2020), PlasticineLab (Huang et al., 2021), DiSECt (Heiden et al., 2021), and DiffSim (Qiao et al., 2020).

Original: Each of these simulators specializes in modeling a single type of deformable object and supports  a narrow range of object-specific tasks.
Revised: However, each simulator specializes in modeling a single type of deformable object and supports only a limited range of object-specific tasks.

Original: For example, PlasticineLab (Hu et al., 2020) and DiSECt (Heiden et al., 2021) model only the elasto-plastic objects, including tasks such as sculpting, rolling, and cutting the deformable objects.
Revised: For example, PlasticineLab (Huang et al., 2021) and DiSECt (Heiden et al., 2021) model only the elasto-plastic objects, including tasks such as sculpting, rolling and cutting the deformable objects.

Original: Since the dynamics of each deformable object are vastly different, the physic engine specialized in modeling one type of deformable objects cannot be easily extended to another.
Revised: Since the dynamics of each deformable object are vastly different, the physic engine specialized in modeling one type of deformable object cannot be easily extended to another.

Original: DaXBench bridges this gap by proposing a set of differentiable benchmarks that cover a wide range of deformable objects and the relevant tasks.
Revised: DaXBench bridges this gap by offering a general-purpose simulation framework that covers a wide range of deformable objects and the relevant tasks.

Original: We provide a fairground for comparing and developing all types of DOM methods, especially the differentiable ones.
Revised: We provide a fairground for comparing and developing all types of DOM methods, especially the differentiable ones.

Original: 2. D EFORMABLE OBJECT MANIPULATION ALGORITHMS
Revised: 2. D EFORMABLE OBJECT MANIPULATION ALGORITHMS

Original: Deformable Object Manipulation (DOM) is more challenging than its rigid body counterpart due to the infinite dimensionality of the state space and the corresponding complex dynamics.

Original: These invalidate the direct application of the existing methods on rigid object manipulation.
Revised: These challenges make it difficult to apply existing methods for rigid object manipulation directly to DOM.

Original:  DOM methods need toexplicitly handle the enormous DoFs in the state space and the complex dynamics.
Revised: Methods designed for DOM must be able to handle a large number of degrees of freedom in the state space and the complexity of the dynamics.

Original: Despite the challenges, many interesting ideas have been proposed for DOM.
Revised: Despite these challenges, many interesting approaches have been proposed for DOM.

Original: Depending on the algorithmic paradigms, we categorizethe DOM methods into three groups: planning, Imitation Learning (IL), and Reinforcement Learning (RL).
Revised: Depending on the algorithmic paradigms, we categorize DOM methods into three groups: planning, Imitation Learning (IL), and Reinforcement Learning (RL).

Original: Our discussion below focuses on the general DOM methods for a standardized set of tasks.
Revised: Our discussion below focuses on the general DOM methods for a standardized set of tasks.

Original: We refer to (Sanchez et al., 2018; Khalil & Payeur, 2010) for a more detailed survey on prior methods for robot manipulation of deformable objects.
Revised: We refer to (Sanchez et al., 2018; Khalil & Payeur, 2010) for a more detailed survey on prior methods for robot manipulation of deformable objects.

Original: Reinforcement Learning algorithms using differentiable physics such as SHAC (Xu et al., 2022) and APG (Freeman et al., 2021) are also explored.
Revised: Reinforcement Learning algorithms using differentiable physics such as SHAC (Xu et al., 2022) and APG (Freeman et al., 2021) are also explored.

Original: These methods use the ground-truth gradients on 
Revised: These methods use the ground-truth gradients on the dynamics to optimize the local actions directly, thereby bypassing the massive sampling for the policy gradient estimation. Imitation Learning is another important learning paradigm for DOM. To overcome the “curse of history”, the existing works such as (Seita et al., 2021; Sundaresan et al., 2020; Ganapathi et al., 2021) overcome the enormous state/action space by learning the low dimensional latent space and

Original: Engine
Revised: Engine

Original: Application
Revised: Application

Original: Differentiability
Revised: Differentiability

Original: Parallelization
Revised: Parallelization

Original: Liquid
Revised: Liquid

Original: Fabric
Revised: Fabric

Original: Elastoplastic
Revised: Elastoplastic

Original: Mixture # of simulations per process
Revised: Mixture # of simulations per process

Original: Multi-Device
Revised: Multi-Device

Original: Training
Revised: Training

Original: PlasticineLab ✗ ✗ ✓ ✗ ✓
Revised: PlasticineLab ✗ ✗ ✓ ✗ ✓

Original: Single ✗
Revised: Single ✗

Original: DiSECt ✗ ✗ ✓ ✗ ✓
Revised: DiSECt ✗ ✗ ✓ ✗ ✓

Original: Single ✗
Revised: Single ✗

Original: DiffSim ✗ ✓ ✗ ✗ ✓
Revised: DiffSim ✗ ✓ ✗ ✗ ✓

Original: Single ✗
Revised: Single ✗

Original: SoftGym ✓ ✓ ✗ ✗ ✗
Revised: SoftGym ✓ ✓ ✗ ✗ ✗

Original: Single ✗
Revised: Single ✗

Original: DaXBench ✓ ✓ ✓ ✓ ✓
Revised: DaXBench ✓ ✓ ✓ ✓ ✓

Original: Multiple✓ the dynamics to optimize the local actions directly, thereby bypassing the massive sampling for the policy gradient estimation. Imitation Learning is another important learning paradigm for DOM.
Revised: Multiple

Original: To overcome the “curse of dimensionality”, the existing works such as (Seita et al., 2020; Sundaresan et al., 2020; Ganapathi et al., 2020) overcome the enormous state/action space by learning the low dimensional latent space and simplifying the primitive actions to only pick-and-place.
Revised: [Table] simplifying the primitive actions to only pick-and-place.

Original: ILD (Chen et al., 2022) is another line of IL method that utilizes the differentiable dynamics to match the entire trajectory all at once; therefore, it reduces the state coverage requirement by the prior works.
Revised: ILD (Chen et al., 2022) is another line of IL method that utilizes the differentiable dynamics to match the entire trajectory all at once; therefore, it reduces the state coverage requirement by the prior works.

Original: Motion planning for DOM has been explored in (Wi et al., 2022;
Revised: Motion planning for DOM has been explored in (Wi et al., 2022;

Original: Shen et al., 2022; Lippi et al., 2020; Nair & Finn, 2019; Yan et al., 2020).
Revised: Shen et al., 2022; Lippi et al., 2020; Nair & Finn, 2019; Yan et al., 2021; Ma et al., 2022).

Original: These methods overcome the enormous DoFs of the original state space by learning a low-dimensional latent state representation and a corresponding dynamic model for planning.
Revised: These methods overcome the enormous DoFs of the original state space by learning a low-dimensional latent state representation and a corresponding dynamic model for planning.

Original: The success of these planning methods critically depends on the dimensionality and quality of the learned latent space and the dynamic model, which in itself is challenging.
Revised: The success of these planning methods critically depends on the dimensionality and quality of the learned latent space and the dynamic model, which in itself is challenging.

Original: We believe that developing a holistic benchmark with its differentiable simulator can provide a fairground to compare with the existing  methods.
Revised: We believe that the development of a general-purpose simulation framework with a differentiable simulator can provide a platform for comparing existing DOM methods.

Original: This allows the researchers to  better understand the progress and limitations of current DOM methods and offers further insights to advance the progress of DOM methods from all paradigms.
Revised: Such a framework enables researchers to gain a better understanding of the progress and limitations of current DOM methods, which in turn can provide valuable insights for advancing DOM methods across all paradigms.

Original:  D A XB ENCH : D EFORMABLE - AND -J AX B ENCHMARK
Revised:  D A XB ENCH : D EFORMABLE - AND -J AX B ENCHMARK

Original: In this work, we introduce DaXBench, a holistic differentiable DOM benchmark that covers a wide range of deformable objects and manipulation tasks.
Revised: DaXBench is a general-purpose differentiable DOM simulation platform that covers a wide range of deformable objects and manipulation tasks.

Original: To benchmark the DOM methods from all paradigms, especially the differentiable ones, we implemented our own differentiable simulator,
Revised: To facilitate benchmarking of DOM methods from all paradigms, particularly differentiable methods, we developed our own differentiable simulator,

Original: Deformable-and-JAX (DaX), which is  highly efficient and parallelizable.
Revised: Deformable-and-JAX (DaX), which is both highly efficient and parallelizable.

Original: In this section, wewill first briefly introduce our simulator DaX. Then we shall focus on introducing the benchmark tasks we  implemented for DaXBench.
Revised: In this section, we first provide a brief overview of our simulator, DaX, and then focus on introducing the benchmark tasks we have implemented for DaXBench.

Original: 3.O UR S IMULATOR : D A X
Revised: 3. D A X

Original: fWe introduce DaX, a differentiable simulator that specializes in modeling various types of deformable object manipulation with high parallelization.
Revised: DaX is a differentiable simulator designed to model various types of deformable object manipulation with high parallelization capabilities.

Original: Comparison with other existing simulators has been shown in Table 1. DaX is implemented using JAX (Bradbury et al., 2018), which supports highly paralleled computation with great automatic differentiation directly optimized on the GPU kernel level.
Revised: A comparison with other existing simulators can be found in Table 1. DaX is implemented using JAX (Bradbury et al., 2018), which supports highly parallelized computation with excellent automatic differentiation directly optimized at the GPU kernel level.

Original: In addition, DaX can connect seamlessly  with the plethora of learning algorithms implemented using JAX, thus enabling the entire computation graph to be highly parallelized end-to-end.
Revised: Moreover, DaX can be seamlessly integrated with the numerous learning algorithms implemented using JAX, allowing for the entire computation graph to be highly parallelized end-to-end.

Original: State Representations and Dynamic Models DaX adopts different state representations and dynamic systems to model the various types of deformable objects with distinct underlying deformation and dynamics.
Revised: State Representations and Dynamic Models DaX adopts different state representations and dynamic systems to model different types of deformable objects with distinct underlying deformation and dynamics.

Original: For liquid and rope that undergo severe deformation, DaX chooses to represent their[Figure] states using particles and use Moving Least Square Material Point Method (MLS-MPM)
Revised: To model liquid and rope, which undergo significant deformation, DaX represents their states using particles and uses Moving Least Square Material Point Method (MLS-MPM)

Original: (Hu et al., 2018a; Sulsky et al., 1994) to model their dynamics on the particle level.
Revised: (Hu et al., 2018; Sulsky et al., 1994) to model their dynamics on the particle level.

Original: On the other hand, for the cloth that only undergoes limited deformation, DaX models its state using mesh and uses the less  expensive mass-spring system to model its dynamics.
Revised: For cloth, which only undergoes limited deformation, DaX models its state using mesh and uses a less computationally expensive mass-spring system to model its dynamics.

Original: With these modeling choices, DaX can strike  a balance betweenthe modeling capacity andits complexity.
Revised: These modeling choices allow DaX to strike [Figure] a balance between modeling capacity and complexity.

Original: We refer the readers to the appendix for more details.
Revised: For more information on the object models used in DaX please refer to Appendix section A.1.

Original: Designed for Customization DaX allows full customization of new tasks based on the existing framework.
Revised: Designed for Customization DaX allows full customization of new tasks based on the existing framework.

Original: Objects with arbitrary shapes and different rigidness can be added to the simulation.
Revised: Objects with arbitrary shapes and different rigidness can be added to the simulator with ease.

Original: Primitive actions can also be customized.
Revised: Moreover, primitive actions can also be customized.

Revised: For a code example of how to customize a new environment in DaX, please refer to Appendix section A.2.

Original: Saving Memory DaX uses two tricks to optimize the efficiency and memory consumption of the end-to-end gradient computation, namely lazy dynamic update and “checkpointing method” (Qiao et al., 2021; Chen et al., 2016; Griewank & Walther, 2000).
Revised: Saving Memory DaX uses two tricks to optimize the efficiency and memory consumption of the end-to-end gradient computation, namely lazy dynamic update and “checkpointing method” (Qiao et al., 2021; Chen et al., 2016; Griewank & Walther, 2000).

Original: To optimize MPM’s time and memory consumption  for a single timestep, DaX lazily updates the dynamic of a small region affected by the manipulation at that step.
Revised: To optimize the time and memory consumption of MPM for a single timestep, DaX lazily updates the dynamic of a small region affected by the manipulation at that step.

Original: In addition, DaX only stores the values of a few states (i.e. checkpoints) at the forward pass; during the backward propagation, DaX re-computes segments of the forward values starting from every stored state inthe reverse order sequentially for the overall gradient.
Revised: Moreover, DaX only stores the values of a few states (i.e., checkpoints) during the forward pass. During the backward propagation, DaX re-computes segments of the forward values starting from every stored state in reverse order sequentially for the overall gradient.

Original: We referthe reader to the appendix for details.
Revised: For more details on these techniques, please refer to Appendix section A.4.

Original: 3.O UR B ENCHMARK T ASKS
Revised: 3. B ENCHMARK T ASKS

Original: DaXBench consists of a wide variety of Deformable Object Manipulation tasks, including manipulating liquid, rope, cloth, and  elastoplastic objects, as illustrated in Figure 2.
Revised: DaXBench provides a comprehensive set of Deformable Object Manipulation tasks, including liquid pouring, rope wiping, cloth folding, and sculpting elastoplastic objects, as illustrated in Figure 2.

Original: In addition to the tasks presented in the previous work, we propose a few new challenging long horizon tasks that require finer-grained control; these new tasks can better benchmark the scalability ofthe DOM methods.
Revised: These tasks are representative of the common deformable objects encountered in daily life. Additionally, we propose several new challenging long-horizon tasks that require finer-grained control, allowing for better benchmarking of the scalability of DOM methods.

Original: The horizon of the task ismainly determined by how many steps it takes human experts to finish the task.
Revised: The horizon of each task is determined by how many steps a human expert takes to finish the task.

Original: DaXBench, therefore, offers a common ground to compare the existing methods and future work.
Revised: DaXBench provides a standardized platform for comparing existing methods and future work on DOM.

Original: In this section, we detail the tasks supported by DaXBench.
Revised: In this section, we provide detailed information on the tasks supported by DaXBench.

Original: Long Horizon Tasks The time complexity for the search-based planning method grows exponentially with the planning horizon, known as  “curse of dimensionality”; this is exacerbated by the large state space of deformable objects.
Revised: Long Horizon Tasks The time complexity for the search-based planning method grows exponentially with the planning horizon, known as the “curse of history”; this is exacerbated by the large state space of deformable objects.

Original: The existing DOM benchmark tasks  have relatively short horizons.
Revised: However, the existing DOM benchmark tasks typically have relatively short horizons.

Original: For example, the cloth folding and unfolding tasks in SoftGym consist of  1-3 steps.
Revised: For example, the cloth folding and unfolding tasks in SoftGym consist of only 1-3 steps.

Original: DaXBench provides tasks with longer horizons.
Revised: DaXBench addresses this limitation by providing tasks with longer horizons, enabling better evaluation of the scalability of search-based planning methods.

Original: Tasks without Macro-Actions A macro-action can be seen as executing raw actions for multiple steps.
Revised: Tasks without Macro-Actions A macro-action can be seen as executing raw actions for multiple steps.

Original: For example, we can define a pick-and-place or push macro-action as a 6-dimensional real vector ( x, y, z, x ′ , y ′ , z ′ ) representing the start ( x, y, z ) and end ( x ′ , y ′ , z ′ ) positions of the gripper, and the raw actions are Cartesian velocities of the gripper.
Revised: For example, we can define a pick-and-place or push macro-action as a 6-dimensional real vector ( x, y, z, x ′ , y ′ , z ′ ) representing the start ( x, y, z ) and end ( x ′ , y ′ , z ′ ) positions of the gripper, and the raw actions are Cartesian velocities of the gripper.

Original: Macro-actions reduce the dimension of the action space and the effective horizon, and hence make scaling easier for the DOM methods.
Revised: Macro-actions reduce the dimension of the action space and the effective horizon, and hence make scaling easier for the DOM methods.

Original: However, macro-actions might be too coarse for some tasks.
Revised: However, macro-actions might be too coarse for some tasks.

Original: Taking whipping a rope to hit a target location as an example, the momentum needs to be adjusted at a relatively high frequency throughout the execution.
Revised: For example, when whipping a rope to hit a target location, the momentum needs to be adjusted at a relatively high frequency throughout the execution, making macro-actions unsuitable.

Original: DaXBench includes tasks that  haveno suitable macro-actions.
Revised: DaXBench includes tasks that do not have suitable macro-actions to provide a more comprehensive evaluation of DOM methods.

Original: 3.2. L IQUID M ANIPULATION
Revised: 3.2. L IQUID M ANIPULATION

Original: Pour-Water Pour a bowl of water to the target bowl as quick as possible.
Revised: Pour-Water Pour a bowl of water into the target bowl as quickly as possible.

Original: The agent directly controls the velocity and rotation (in Cartesian space) of the gripper that holds the bowl.
Revised: The agent directly controls the velocity and rotation (in Cartesian space) of the gripper that holds the bowl.

Original: In particular, the action space of the agent is a 4-dimensional vector a =
Revised: In particular, the action space of the agent is a 4-dimensional vector a =

Original: ( v x , v y , v z , w ) , for which ( v x , v y , v z ) represents the linear velocity of the gripper, and w denotes the angular velocity around the wrist.
Revised: ( v x , v y , v z , w ) , for which ( v x , v y , v z ) represents the linear velocity of the gripper, and w denotes the angular velocity around the wrist.

Original: Pour-Soup Pouring a bowl of soup with various solid ingredients to the target bowl as fast as possible.
Revised: Pour-Soup Pour a bowl of soup with various solid ingredients into the target bowl as fast as possible.

Original: The actions are the same as those of Pour-Water.
Revised: The actions are the same as those of Pour-Water.

Original: This task is more challenging than Pour-Water due to the additional interactions between the liquid and the solid ingredients.
Revised: This task is more challenging than Pour-Water due to the additional interactions between the liquid and the solid ingredients.

Original: In particular, the agent needs to promptly react when a solid ingredient is transferred, so as to 1) adjust for the sudden decrease in load on the gripper and 2) control the momentum of the ingredient when hitting the soup surface to minimize spilling.
Revised: In particular, the agent needs to promptly react when a solid ingredient is transferred, so as to 1) adjust for the sudden decrease in load on the gripper and 2) control the momentum of the ingredient when hitting the soup surface to minimize spilling.

Original: Neither tasks have suitable high-level macro-actions and the task horizons are 100 steps.
Revised: Neither tasks have suitable high-level macro-actions and the task horizons are 100 steps

Original:  3.2. R OPE M ANIPULATION
Revised: . 3.2. R OPE M ANIPULATION

Original: Push-Rope Push the rope to the pre-specified configuration as quickly as possible.
Revised: Push-Rope Push the rope to the pre-specified configuration as quickly as possible.

Original: We randomly initialize the configuration of the rope.
Revised: We randomly initialize the configuration of the rope.

Original: The agent uses a single gripper to execute the push macroactions.
Revised: The agent uses a single gripper to execute the push macroactions.

Original: The horizon for this task is 6 steps.
Revised: The horizon for this task is 6 steps.

Original: Whip-Rope Whip the rope into a target configuration.
Revised: Whip-Rope Whip the rope into a target configuration.

Original: The agent holds one end of the rope and controls the gripper velocity to whip the rope.
Revised: The agent holds one end of the rope and controls the gripper’s velocity to whip the rope.

Original: In particular, the action space of this task is a 3dimensional vector a
Revised: In particular, the action space of this task is a 3dimensional vector a

Original: = ( v x , v y , v z ) that denotes the velocity of the gripper in the Cartesian space.
Revised: = ( v x , v y , v z ) that denotes the velocity of the gripper in the Cartesian space.

Original: This task cannot be completed by the abstract macro-actions since the momentum and the deformation of the rope has to be adjusted at a relatively high frequency.
Revised: This task cannot be completed by the abstract macro-actions since the momentum and the deformation of the rope has to be adjusted at a relatively high frequency.

Original: The task horizon is 70 steps.
Revised: The task horizon is 70 steps.

Original: 3.2. C LOTH M ANIPULATION
Revised: 3.2. C LOTH M ANIPULATION

Original: Fold-Cloth Fold a piece of flattened cloth and move it to a target location.
Revised: Fold-Cloth Fold a piece of flattened cloth and move it to a target location.

Original: This task has two variants: the easy version requires the agent to execute 1 fold and the difficult version 3 folds.
Revised: This task has two variants: the easy version requires the agent to execute 1 fold and the difficult version 3 folds.

Original: For both variants, the task begins with the cloth lying flat on the table at an arbitrary position.
Revised: For both variants, the task begins with the cloth lying flat on the table at an arbitrary position.

Original: The agent executes pick-and-place macro-actions.
Revised: The agent executes pick-and-place macro-actions.

Original: The task horizon is 3 and 4 for the easy and difficult versions respectively.
Revised: The task horizon is 3 and 4 for the easy and difficult versions respectively.

Original: Fold-T-shirt Fold apiece of T-shirt to a target location.
Revised: Fold-T-shirt Fold a T-shirt to a target location.

Original: The task begins with the T-shirt lying flat on the table at an arbitrary position.
Revised: The task begins with the T-shirt lying flat on the table at an arbitrary position.

Original: The agent executes pick-and-place macro-actions.
Revised: The agent executes pick-and-place macro-actions.

Original: The task horizon is 4.
Revised: The task horizon is 4.

Original: Unfold-Cloth Flatten a piece of folded cloth to a target location as fast as possible.
Revised: Unfold-Cloth Flatten a piece of folded cloth to a target location as fast as possible.

Original: This task also has two variants that differ in the initial configurations of the folded cloth: the cloth in the easy version has been folded once and that in the difficult version has been folded three times.
Revised: This task also has two variants that differ in the initial configurations of the folded cloth: the cloth in the easy version has been folded once and that in the difficult version has been folded three times.

Original: This task has the same action space as the Fold-Cloth task.
Revised: This task has the same action space as the Fold-Cloth task.

Original: The task horizon is 10 steps for both variants.
Revised: The task horizon is 10 steps for both variants.

Original: All of these tasks share similar APIs to the OpenAI Gym’s standard environments.
Revised: All of these tasks share similar APIs to the OpenAI Gym’s standard environments.

Original: We provide a general template to define tasks in our simulator.
Revised: We provide a general template to define tasks in our simulator.

Original: This not only better organizes the existing task environments, but also allows the users to customize the existing task environments to better suit their research requirements.
Revised: This not only better organizes the existing task environments but also allows the users to customize the existing task environments to better suit their research requirements.

Original: The variables and constants defined in the template are easily interpretable quantities that correspond to real physical semantics.
Revised: The variables and constants defined in the template are easily interpretable quantities that correspond to real physical semantics.

Original: This allows the users to better understand the environment, which can also help with the development and debugging process.
Revised: This allows the users to better understand the environment, which can also help with the development and debugging process.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we aim to answer the following two questions: 1) Are the existing representative methods capable of solving the tasks in DaXBench? 2)
Revised: In this section, we aim to answer the following two questions: 1) Are the existing representative methods capable of solving the tasks in DaXBench? 2)

Original: Within each paradigm, will the differentiable simulator help in completing the tasks?
Revised: Within each paradigm, will the differentiable simulator help in completing the tasks?

Original: To investigate these two questions, we benchmark eight representative methods of deformable object manipulation, covering planning, imitation learning, and reinforcement learning.
Revised: To investigate these two questions, we benchmark eight representative methods of deformable object manipulation, covering planning, imitation learning, and reinforcement learning.

Original: Each paradigm has different assumptions and requirements for input knowledge.
Revised: Each paradigm has different assumptions and requirements for input knowledge.

Original: We summarize the baseline methods and the relevant information in Table 2.
Revised: We summarize the baseline methods and the relevant information in Table 2.

Original: 4. E XPERIMENTAL S ETUP
Revised: [Figure] Reinforcement

Original: We report the mean and variance of the performance for each task over 5 different seeds.

Original: [Figure]
Revised: [Figure]

Original: Imitation
Revised: Imitation

Original: [Figure]
Revised: [Figure]

Revised: Planning

Original: ILD
Revised: Methods PPO SHAC APG

Original: [Figure]
Revised: [Figure]

Revised: ILD

Revised: CEM

Original: # expert
Revised: [Figure] MPC diff [Figure] diff

Original: [Figure]
Revised: [Figure]

Revised: # expert demos — — —  — — —

Revised: Reward

Original: [Figure]
Revised: [Figure]

Original: ✓ ✓ ✓ — — ✓ ✓ ✓
Revised: ✓ ✓ ✓ — — ✓ ✓ ✓

Original: Differentiability ✗ ✓ ✓ ✗ ✓ ✗ ✓ ✓
Revised: Differentiability ✗ ✓ ✓ ✗ ✓ ✗ ✓ ✓

Original: [Figure]
Revised: [Figure]

Original: 1, with larger values being closer to the goal state.
Revised: , with larger values being closer to the goal state.

Original: The x-axis is the number of training iterations.
Revised: The x-axis is the number of training iterations.

Original: All methods are trained in the same number of batched environments.
Revised: All methods are trained in the same number of batched environments.

Revised: We report the mean and variance of the performance for 5 seeds and each seed has 20 rollouts with different initializations.

Original: We refer the readers to Table 4 for the numerical results in the appendix.
Revised: We refer the readers to Table 4 for the numerical results in the appendix.

Revised: 4. E XPERIMENTAL S ETUP

Original: Reward function For each task, the goal is specified by the desired final positions of the set of particles, g , representing the deformable object.
Revised: Reward function For each task, the goal is specified by the desired final positions of the set of particles, g , representing the deformable object.

Original: In this setup, the reward can be intuitively defined by how well the current object particles match with those of the goal.
Revised: In this setup, the reward can be intuitively defined by how well the current object particles match with those of the goal.

Original: Hence, we define the groundtruth reward as r gt ( s, a ) =
Revised: Hence, we define the groundtruth reward as r gt ( s, a ) =

Original: exp( − λD ( s ′ , g )) , where s ′ is the next state resulted by a at the current state s , and D ( s, g ) is a non-negative distance measure between the positions of  object’s particles s and those of the goal g .
Revised: exp( − λD ( s ′ , g )) , where s ′ is the next state resulted from a at the current state s , and D ( s, g ) is a non-negative distance measure between the positions of the object’s particles s and those of the goal g .

Original: However, using the ground-truth reward along is inefficient for learning.
Revised: However, using the ground-truth reward alone is inefficient for learning.

Original: The robot’s end-effector can only change the current object pose by contacting it.
Revised: The robot’s end-effector can only change the current object pose by contacting it.

Original: Therefore, to most efficiently manipulate the object to the desired configuration, we add an auxiliary reward to encourage the robot to contact the object.
Revised: Therefore, to most efficiently manipulate the object to the desired configuration, we add an auxiliary reward to encourage the robot to contact the object.

Original: We define the auxiliary reward as r aux
Revised: We define the auxiliary reward as r aux

Original: ( s, a ) =
Revised: ( s, a ) =

Original: exp( − L 2 ( s, a )) , where L 2 ( s, a ) measures the L 2 distance between the end-effector and the object.
Revised: exp( − L 2 ( s, a )) , where L 2 ( s, a ) measures the L 2 distance between the end-effector and the object.

Original: During training, we use  sum of the groundtruth and auxiliary reward functions r gt + r aux , and during evaluation we only use the ground-truth reward r gt .
Revised: During training, we use the sum of the groundtruth and auxiliary reward functions r gt + r aux , and during evaluation, we only use the ground-truth reward r gt .

Original: 4. R EINFORCEMENT L EARNING
Revised: 4. R EINFORCEMENT L EARNING

Original: Methods.
Revised: Methods.

Original: We benchmark PPO (Schulman et al., 2017), SHAC (Xu et al., 2022), and APG (Freeman et al., 2021) as the representative RL methods.
Revised: We benchmark PPO (Schulman et al., 2017), SHAC (Xu et al., 2022), and APG (Freeman et al., 2021) as the representative RL methods.

Original: PPO is a widely used model-free nondifferentiable RL method with competitive performance, while SHAC and APG are the two latest
Revised: PPO is a widely used model-free nondifferentiable RL method with competitive performance, while SHAC and APG are the two latest

Original: RL methods that utilize the differentiable simulator.
Revised: RL methods that utilize the differentiable simulator.

Original: The computation graph of SHAC is illustrated in Figure 4a.
Revised: The computation graph of SHAC is illustrated in Figure 4a.

Original: By comparing the performance of the DaXBench tasks across these three methods, we aim to further study the benefits and limitations of applying the differentiable physics to RL.
Revised: By comparing the performance of the DaXBench tasks across these three methods, we aim to further study the benefits and limitations of applying the differentiable physics to RL.

Original: Overall Performance.
Revised: Overall Performance.

Original: The learning curves of RL methods are shown in Figure 3.
Revised: The learning curves of RL methods are shown in Figure 3.

Original: The performance difference between PPO and the differentiable RL methods shows opposite trends in the high-level macro-action-based and local-level control tasks.
Revised: The performance difference between PPO and the differentiable RL methods shows opposite trends in the high-level macro-action-based and local-level control tasks.

Original: To our surprise, for most of the short-horizon  macro-action-based tasks, PPO performs consistently better than the differentiable RL methods.
Revised: To our surprise, for most of the short-horizon [Table] macro-action-based tasks , PPO performs consistently better than the differentiable RL methods.

Original: This seems to contradict the experiment results of SHAC (Xu et al., 2022), which shows stronger performance than PPO in its own simulator on a wide range of context-rich MuJoCo tasks.How- [Table]
Revised: This seems to contradict the experiment results of SHAC (Xu et al., 2022), which shows stronger performance than PPO in its own simulator on a wide range of context-rich MuJoCo tasks.

Original: ever, for long-horizon tasks with low-level controls, the differentiable RL methods, especially APG, outperform PPO by a large margin.
Revised: However, for long-horizon tasks with low-level controls, the differentiable RL methods, especially APG, outperform PPO by a large margin.

Original: Main Challenge for the differentiable-physics-based RL methods: Exploration.
Revised: Main Challenge for the differentiable-physics-based RL methods: Exploration.

Original: We argue that, on high-level macro-action-based tasks, the main limiting factor of the performance of differentiable-physics-based RL is the lack of exploration during training.
Revised: We argue that, on high-level macro-action-based tasks, the main limiting factor of the performance of differentiable-physics-based RL is the lack of exploration during training.

Original: The need to balance this trade-off is exaggerated by the enormous DoFs and the complex dynamics unique to the DOM tasks.
Revised: The need to balance this trade-off is exaggerated by the enormous DoFs and the complex dynamics unique to the DOM tasks.

Original: Especially for the high-level macro-action-based tasks, their non-smooth/discontinuous/nonconvex optimization landscape necessitates using a good exploration strategy.
Revised: Especially for the high-level macro-action-based tasks, their non-smooth/discontinuous/nonconvex optimization landscape necessitates using a good exploration strategy.

Original: However, the existing differentiable-physics-based RL methods rely exclusively on the simulation gradient to optimize the policy.
Revised: However, the existing differentiable-physics-based RL methods rely exclusively on the simulation gradient to optimize the policy.

Original: They are not entropy regularized; hence, they may quickly collapse to the local optimal.
Revised: They are not entropy regularized; hence, they may quickly collapse to the local optimal.

Original: Without explicit exploration, the differentiable-physics-based RL methods fail to reach the near-optimal state using the simulation gradients alone.
Revised: Without explicit exploration, the differentiable-physics-based RL methods fail to reach the near-optimal state using the simulation gradients alone.

Original: Taking Fold-Cloth-3 as an example, if the randomly initialized policy never touches the cloth, the local simulation gradients cannot guide the agent to touch the cloth since the rewards and, consequently, the gradients for all not-in-contact actions are zero.
Revised: Taking Fold-Cloth-3 as an example, if the randomly initialized policy never touches the cloth, the local simulation gradients cannot guide the agent to touch the cloth since the rewards and, consequently, the gradients for all not-in-contact actions are zero.

Original: Differentiable-physics-based RL are sensitive to the Optimization Landscape.
Revised: Differentiable-physics-based RL are sensitive to the Optimization Landscape.

Original: Differentiablephysics-based RL can largely improve the performance of some low-level control tasks if the optimization landscape is smooth and convex.
Revised: Differentiablephysics-based RL can largely improve the performance of some low-level control tasks if the optimization landscape is smooth and convex.

Original: Taking Whip-Rope as an example, the optimal low-level control sequence is a smooth motion trajectory that lies on a relatively smooth and well-behaved plane for the gradient-based methods.
Revised: Taking Whip-Rope as an example, the optimal low-level control sequence is a smooth motion trajectory that lies on a relatively smooth and well-behaved plane for the gradient-based methods.

Original: This explains why APG outperforms the non-differentiablephysics-based PPO by a large margin.
Revised: This explains why APG outperforms the non-differentiablephysics-based PPO by a large margin.

Original: 4. I MITATION L EARNING
Revised: 4. I MITATION L EARNING

Original: Methods.
Revised: Methods.

Original: We benchmark Transporter (Seita et al., 2020) and ILD (Chen et al., 2022) as the two representative Imitation Learning DOM methods.
Revised: We benchmark Transporter (Seita et al., 2021) and ILD (Chen et al., 2022) as the two representative Imitation Learning DOM methods.

Original: Transporter is a popular DOM method that is wellknown for its simple implementation and competitive performance.
Revised: Transporter is a popular DOM method that is well-known for its simple implementation and competitive performance.

Revised: We use particles as input to evaluate the performance of Transporter in our experiment for a fair comparison.

Revised: The original implementation of Transporter uses RGBD as input.

Revised: We refer the readers to the Appendix section D for the performance of the Transporter with RGB channels.

Original: ILD is the latest differentiablephysics-based IL method; it utilizes the simulation gradients to reduce the required number of expert demonstrations and to improvethe performance.
Revised: ILD is the latest differentiablephysics-based IL method; it utilizes the simulation gradients to reduce the required number of expert demonstrations and to improve performance.

Original: Note that since Transporter abstracts its action space to pick-and-place actions to simplify the learning, it cannot be extended to the low-level control tasks.
Revised: Note that since Transporter abstracts its action space to pick-and-place actions to simplify the learning, it cannot be extended to the low-level control tasks.

Original: Our comparison for the IL methods will first focus on the high-level macro-actionbased tasks, then we will analyze the performance of ILD with the expert.
Revised: Our comparison of the IL methods will first focus on the high-level macro-action-based tasks, then we will analyze the performance of ILD with the expert.

Original: Overall performance For the high-level macro-action-based tasks, ILD outperforms Transporter by a large margin for all tasks.
Revised: Overall performance For the high-level macro-action-based tasks, ILD outperforms Transporter by a large margin for all tasks.

Original: For example, ILD triples the performance Transporter in Fold-Clothand doubles it in Fold-Cloth-3 and Unfold-Cloth-1.
Revised: For example, ILD outperforms Transporter in Fold-Cloth-1, Fold-Cloth3, and Unfold-Cloth-1.

Original: We highlight that this significant performance gain is achieved by using a much smaller number of expert demonstrations.
Revised: We highlight that this significant performance gain is achieved by using a much smaller number of expert demonstrations.

Original: We attribute this improvement to the additional information provided by the simulation gradients, where now ILD can reason over the ground-truth  dynamics to bring the trajectory closer to the expert, at least locally.
Revised: We attribute this improvement to the additional information provided by the simulation gradients, where now ILD can reason over the ground-truth [Figure] dynamics to bring the trajectory closer to the expert, at least locally.

Original: As illustrated in Figure 4, the gradients used to optimize each action comes from the globally optimal expert demonstrations 1) at the current step and 2) in all the steps after.
Revised: As illustrated in Figure 4, the gradients used to optimize each action comes from the globally optimal expert demonstrations 1) at the current step and 2) in all the steps after.

Original: In contrast, the gradients used in SHAC only count the reward at the current step.
Revised: In contrast, the gradients used in SHAC only count the reward at the current step.

Original: The differentiable RL faces the challenge ofthe exploding/vanishing gradients and lack of exploration.
Revised: The differentiable RL faces the challenge of exploding/vanishing gradients and lack of exploration.

Original: Differentiable IL alleviates these problems by having step-wise, globally optimal guidance from expert demonstrations.
Revised: Differentiable IL alleviates these problems by having step-wise, globally optimal guidance from expert demonstrations.

Original: [Figure]

Original: Challenge for ILD: Unbalanced State Representations.
Revised: Challenge for ILD: Unbalanced State Representations.

Original: ILD does not perform well on tasks like
Revised: ILD does not perform well on tasks like

Original: Pour-Water and Pour-Soup with unbalanced state representations, for example, 1000 × 3 features of particles and 6 gripper states.
Revised: Pour-Water and Pour-Soup with unbalanced state representations, for example, 1000 × 3 features of particles and 6 gripper states.

Original: The learned signals from the particles overwhelm the signals from the gripper states.
Revised: The learned signals from the particles overwhelm the signals from the gripper states.

Original: Therefore, ILD cannot learn useful strategies with unbalanced learning signals.
Revised: Therefore, ILD cannot learn useful strategies with unbalanced learning signals.

Original: Softgym (Lin et al., 2020) also concurs with the results that policies with reduced state representation perform better than those with  full state.
Revised: Softgym (Lin et al., 2021) also concurs with the results that policies with reduced state representation perform better than those with the full state.

Original: A more representative representation of the states needs to be considered and we leave it for future study.
Revised: A more representative representation of the states needs to be considered and we leave it for future study.

Original: In conclusion, differentiable methods can optimize the policies efficiently if the optimization landscape is well-behaved.
Revised: In conclusion, differentiable methods can optimize the policies efficiently if the optimization landscape is well-behaved.

Original: Forthe tasks that require tremendous explorations or with sparse rewards, gradient-based methods suffer severely from the local minima and gradient instability issues.
Revised: For tasks that require tremendous explorations or with sparse rewards, gradient-based methods suffer severely from the local minima and gradient instability issues.

Original: 4. P LANNING
Revised: 4. P LANNING

Original: Methods.
Revised: Methods.

Original: We benchmark the classic CEM-MPC (Richards, 2005) as a representative random shooting method that uses a gradient-free method to optimize the action sequences for the highest final reward.
Revised: We benchmark the classic CEM-MPC (Richards, 2005) as a representative random shooting method that uses a gradient-free method to optimize the action sequences for the highest final reward.

Original: To study the effect of using differentiable physics, we implement two versions of differentiable MPC.
Revised: To study the effect of using differentiable physics, we implement two versions of differentiable MPC.

Original: The basic computation graph for both is illustrated in Figure 4c.
Revised: The basic computation graph for both is illustrated in Figure 4c.

Original: In particular, both differentiable MPC baselines maximize the immediate reward of each action using differentiable physics.
Revised: In particular, both differentiable MPC baselines maximize the immediate reward of each action using differentiable physics.

Original: The difference lies in the initial action sequences: diff-MPC is initialized with an random action sequence, while diff-CEM-MPC with an action sequence optimized by CEM-MPC.
Revised: The difference lies in the initial action sequences: diff-MPC is initialized with a random action sequence, while diff-CEM-MPC with an action sequence optimized by CEM-MPC.

Original: Overall Performance.
Revised: Overall Performance.

Original: The diff-CEM-MPC performs consistently better than or on par with CEMMPC, while Diff-MPC fails to plan any reasonable control sequences for most of  tasks.
Revised: The diff-CEM-MPC performs consistently better than or on par with CEMMPC, while Diff-MPC fails to plan any reasonable control sequences for most of the tasks.

Original: Simulation Gradients as Additional Optimization Signals.
Revised: Simulation Gradients as Additional Optimization Signals.

Original: The vastly different performance between the two versions of the differentiable baselines brings us insights on howthe differentiable physics helps in planning.
Revised: The vastly different performance between the two versions of the differentiable baselines brings us insights into how differentiable physics helps in planning.

Original: Simulation gradients providethe additional signals in the direction to optimize the actions, thereby reducing the sample complexity.
Revised: Simulation gradients provide additional signals in the direction to optimize the actions, thereby reducing the sample complexity.

Original: However, similar to any gradient-based optimization methods, differentiable planning methods are sensitive to the non-convex/non-smooth optimization landscape and face a major limitation of being stuck at the local optimal solution.
Revised: However, similar to any gradient-based optimization methods, differentiable planning methods are sensitive to the non-convex/non-smooth optimization landscape and face a major limitation of being stuck at the local optimal solution.

Original: DOM’s enormous DoFs of DOM and complex dynamics exacerbate these challenges for the differentiable DOM planning methods.
Revised: DOM’s enormous DoFs of DOM and complex dynamics exacerbate these challenges for the differentiable DOM planning methods.

Original: Differentiable Planning needs Good Initialization.
Revised: Differentiable Planning needs Good Initialization.

Original: We attribute the poor performance of DiffMPC to its random control sequence initialization.
Revised: We attribute the poor performance of DiffMPC to its random control sequence initialization.

Original: In particular, this control sequence may significantly deviate from the optimal one over the non-smooth/non-convex landscape, such that the simulation gradients alone cannot bring the control sequence back to the optimal.
Revised: In particular, this control sequence may significantly deviate from the optimal one over the non-smooth/non-convex landscape, such that the simulation gradients alone cannot bring the control sequence back to the optimal.

Original: In contrast, DiffCEM-MPC is initialized with the optimized control sequence by CEM-MPC, which relies on iterative sampling followed by elite selection to escape the local optimality.
Revised: In contrast, DiffCEM-MPC is initialized with the optimized control sequence by CEM-MPC, which relies on iterative sampling followed by elite selection to escape the local optimality.

Original: Hence, the control sequence fed into the differentiable step is already sufficiently close to the globally optimal solution.
Revised: Hence, the control sequence fed into the differentiable step is already sufficiently close to the globally optimal solution.

Original: Thesimulation gradients help diff-CEM-MPC to optimize the action sequences further locally, which explains the performance gain. To summarize, the differentiable simulator can further optimize the near-optimal control sequences locally with much low sample complexity. Yet, its performance critically relies on a good initialization and a locally smooth optimization landscape.
Revised: The

Original: [Figure]
Revised: [Figure]

Original: Differentiable-physics-based methods fundamentally use the simulation gradients as additional signals to optimize the policy/control sequence. Therefore, these methods inherit the generic benefits of additional ground-truth gradients, such as faster convergence and more stable optimization under mild conditions.
Revised: simulation gradients help diff-CEM-MPC to optimize the action sequences further locally, which explains the performance gain. To summarize, the differentiable simulator can further optimize the near-optimal control sequences locally with much low sample complexity.

Original: At the same time, the problems borne with the gradient-based methods are exacerbated in differentiable DOM methods. For example, the performanceof the differentiable methods critically relies on  good initialization and a well-behaved optimization landscape.Without these assumptions, additional effort is required to sufficiently explore the enormous state space and assist the simulation gradient-based optimization in escaping its local minima. We believe more research into the exploration strategy for differentiable DOM methods is necessary to further advance the progress in deformable object manipulation.
Revised: Yet, its performance critically relies on a good initialization and a locally smooth optimization landscape.

Original: 4. S IM -2-R EAL G AP
Revised: 4. S IM -2-R EAL G AP

Original: Granted that the physical engine of DaX is built upon  state-of-the-art simulation method, the fidelity and correctness still need to be tested.
Revised: Granted that the physical engine of DaX is built upon a state-of-the-art simulation method, the fidelity and correctness still need to be tested.

Original: To testify to the correctness of our simulated dynamics tothe real-world physics, we carry out a real robot experiment on a Kinova Gen 3 robot.
Revised: To testify to the correctness of our simulated dynamics to real-world physics, we carry out a real robot experiment on a Kinova Gen 3 robot.

Original: We deploy
Revised: We deploy

Original: CEM-MPC with DaX as the predictor model to a Push-Rope task, as shown in Figure 5.
Revised: CEM-MPC with DaX as the predictor model to a Push-Rope task, as shown in Figure 5.

Original: Our study finds that the resultant state trajectories are similar in simulation and in reality.
Revised: Our study finds that the resultant state trajectories are similar in simulation and in reality.

Original: This validates the fidelity of the tasks implemented based on our engine, and the performance of the DOM methods in our simulator provides hints about their performance on the real robot.
Revised: This validates the fidelity of the tasks implemented based on our engine, and the performance of the DOM methods in our simulator provides hints about their performance on the real robot.

Original: We refer the reader to the appendix for details.
Revised: We refer the reader to the appendix for details.

Original: In addition, the differentiability of DaX enables system identification that can reduces the sim-2-real gap (Le Lidec et al., 2021; Murthy et al., 2021).
Revised: In addition, the differentiability of DaX enables system identification that can reduce the sim-2-real gap (Le Lidec et al., 2021; Murthy et al., 2021).

Original: We can get an error term by comparing the observations in simulation and in reality; by differentiating this error with respect to the simulator parameters, we get gradients for parameter tuning.
Revised: We can get an error term by comparing the observations in simulation and in reality; by differentiating this error with respect to the simulator parameters, we get gradients for parameter tuning.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We introduced DaXBench, a  differentiable benchmark for various deformable object manipulation tasks.
Revised: We introduced DaXBench, a general-purpose differentiable simulation framework for various deformable object manipulation tasks.

Original: To our knowledge, DaXBench is the first benchmark that simulates liquid, rope, fabric, and elastoplastic materials while being differentiable and highly parallelizable.
Revised: To our knowledge, DaXBench is the first platform that simulates liquid, rope, fabric, and elastoplastic materials while being differentiable and highly parallelizable.

Original: Our task coverage goes beyond the common DOM tasks; we innovate several novel tasks, such as WipeRope and PourSoup, to study the performance ofthe DOM methods on long-horizon tasks with low-level controls.
Revised: In addition to common DOM tasks, we also proposed several novel tasks, including WhipRope and PourSoup, to study the performance of DOM methods on long-horizon tasks with low-level controls.

Original: Ourrich task coverage provides a fairground to comparethe existing DOM methods from all paradigms; further, our benchmark provides insights into how the  differentiable-physics-based methodshelp in DOM and the caveats of using this new tool.
Revised: Our task coverage provides a comprehensive platform to compare existing DOM methods from all paradigms, and our empirical study sheds light on the benefits and limitations of differentiable-physics-based methods in DOM.

Original: We believe  DaXBenchcanhelp assist with the algorithmic development of DOM methods, especially the differentiable-physics-based ones of all paradigms. 
Revised: We believe that DaXBenchcan assist in the algorithmic development of DOM methods, particularly those based on differentiable physics, across all paradigms. A CKNOWLEDGEMENTS

Revised: This research is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award

Revised: No: AISG2-PhD-2021-08-015T, AISG2-PhD-2022-01-036T, and

Revised: AISG2-PhD-2021-08-014) and A ∗ STAR under the National Robotics Program (Grant No. 19200054).

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: DaXBench provides a benchmark for deformable object manipulation (DOM).
Revised: DaXBench provides a benchmark for deformable object manipulation (DOM).

Original: We believe that
Revised: We believe that

Original: DaXBench and its simulator DaX have no potential negative societal impacts.
Revised: DaXBench and its simulator DaX have no potential negative societal impacts.

Original: In addition, our work is on benchmarking the DOM methods in simulator, so we do not collect any data or conduct experiments with human subjects.
Revised: In addition, our work is on benchmarking the DOM methods in simulator, so we do not collect any data or conduct experiments with human subjects.

Original: In summary, we have read the ICLR Code of Ethics and ensured that our work conforms to them
Revised: In summary, we have read the ICLR Code of Ethics and ensured that our work conforms to them

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: In this paper all experiments are averaged over 5 random seeds.
Revised: In this paper imitation learning and planning experiments are averaged over 20 random rollouts and

Revised: RL approaches use 100 rollouts.

Original: We have included our source code as an easy-to-install package in the supplementary material.
Revised: We have included our source code as an easy-to-install package in the supplementary material.

Original: A I MPLEMENTATION D ETAILS A. O BJECTS
Revised: A I MPLEMENTATION D ETAILS A. O BJECTS

Original: DaXBench models a variety of deformable objects of different dimensionalities, ranging from 0dimensional liquid, 1-dimensional rope, to 2-dimensional cloth.
Revised: DaXBench models a variety of deformable objects of different dimensionalities, ranging from 0dimensional liquid, 1-dimensional rope, to 2-dimensional cloth.

Original: The underlying deformation and the dynamics vary distinctly for each dimension.
Revised: The underlying deformation and the dynamics vary distinctly for each dimension.

Original: In general, the lower the dimension is, the higher the degrees of freedom (DoFs) of the system, thus the more complex modeling that system becomes.
Revised: In general, the lower the dimension is, the higher the degrees of freedom (DoFs) of the system, thus the more complex modeling that system becomes.

Original: We use two different representations to model the deformable objects to balance the modeling capacity with its complexity: water and rope are modeled using Material Point Method (MPM), while the cloth is modelled using a mass-spring system.
Revised: We use two different representations to model the deformable objects to balance the modeling capacity with its complexity: water and rope are modeled using Material Point Method (MPM), while the cloth is modeled using a mass-spring system.

Original: Modelling Water and Rope using MPM Material Point Method (MPM) demonstrates strong modeling capacity and flexibility.
Revised: Modelling Water and Rope using MPM Material Point Method (MPM) demonstrates strong modeling capacity and flexibility.

Original: MPM models the deformable objects as a set of particles; each particle is represented by the position and some properties such as mass.
Revised: MPM models the deformable objects as a set of particles; each particle is represented by its position and some properties such as mass.

Original: The dynamics and the deformation of the object are computed by simulating the interactions among all individual particles, conforming to some pre-defined physics rules.
Revised: The dynamics and the deformation of the object are computed by simulating the interactions among all individual particles, conforming to some pre-defined physics rules.

Original: This Particle-level representation and the dynamics simulation allow MPM to model the object under severe deformation.
Revised: This Particle-level representation and the dynamics simulation allow MPM to model the object under severe deformation.

Original: We therefore choose to use MPM to model water and rope that can undergo severe deformation, for they can exploit the model capacity of MPM to its full potential.
Revised: We, therefore, choose to use MPM to model water and rope that can undergo severe deformation, for they can exploit the model capacity of MPM to its full potential.

Original: Modelling Cloth using Mass-Spring System Cloth only undergoes limited deformation as it has to conform to the canonical shape of the initial cloth throughthe manipulation.
Revised: Modelling Cloth using Mass-Spring System Cloth only undergoes limited deformation as it has to conform to the canonical shape of the initial cloth through manipulation.

Original: This limited deformable does not require the strong modeling capacity of MPM, which is computationally expensive.
Revised: This limited deformable does not require the strong modeling capacity of MPM, which is computationally expensive.

Original: Therefore, rather than modeling the deformation on the Particle-level, we choose to model the cloth using the mass-spring system, which is a simpler model that uses the physical constraints as a prior.
Revised: Therefore, rather than modeling the deformation on the Particle-level, we choose to model the cloth using the mass-spring system, which is a simpler model that uses the physical constraints as a prior.

Original: A. C ODE E
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 1 import numpy as np 2 fromcore.envs.registration import env_functions
Revised: 1 import numpy as np 2 from

Revised: .envs.registration import env_functions

Original: [Table]

Original: =1) .action_size)) 8 obs, reward, done, info = env.step_diff(actions, state)
Revised: .action_size)) 8 obs, reward, done, info = env.step_diff(actions, state)

Original: Listing 1: Example of Running Cloth Environment 1 import jax 2 import numpy as np 3 from core.envs.registration import env_functions 4 5 env = env_functions["fold_cloth3"](batch_size=32, aux_reward=True) 6 obs, state = env.reset(env.simulator.key_global) 7 actions =
Revised: Listing 1: Example of Running Cloth Environment 1 import jax 2 import numpy as np 3 from core.envs.registration import env_functions 4 5 env = env_functions["fold_cloth3"](batch_size=32, aux_reward=True) 6 obs, state = env.reset(env.simulator.key_global) 7 actions =

Original: np.random.uniform(0, 1, (env.batch_size, env.action_size)) 8 9 @jax.jit 10 @jax.grad 11 def compute_grad(actions, state): obs, reward, done, info = env.step_diff(actions, state) return -reward.sum() 14 15 print("action gradients", compute_grad(actions, state))
Revised: np.random.uniform(0, 1, (env.batch_size, env.action_size)) 8 9 @jax.jit 10 @jax.grad 11 def compute_grad(actions, state): obs, reward, done, info = env.step_diff(actions, state) return -reward.sum() 14 15 print("action gradients", compute_grad(actions, state))

Original: Listing 2: Example of Computing Action Gradients via DaX
Revised: Listing 2: Example of Computing Action Gradients via DaX

Original: A. J AX TO PARALLELIZE
Revised: A. J AX TO PARALLELIZE

Original: One major limitation of the existing simulators is the lack of support for highly paralleled sampling.
Revised: One major limitation of the existing simulators is the lack of support for highly paralleled sampling.

Original: The key challenges for deformable Object Manipulation is on the complex governing equations 
Revised: The key challenges for Deformable Object Manipulation are the complex governing equations and

Original: [Figure]
Revised: [Figure]

Original: and the enormous state space.
Revised: the enormous state space.

Original: The complex governing equations invalidate most of the analytical approaches since solving the complex governing equations are intractable.
Revised: The complex governing equations invalidate most of the analytical approaches since solving the complex governing equations is intractable.

Original: On the other hand, the success of the sampling-based methods hinge on the reasonable coverage of the state space.
Revised: On the other hand, the success of the sampling-based methods hinges on the reasonable coverage of the state space.

Original: The lack of support for highly paralleled sampling to better cover the state space is a major impediment for the works on Deformable Object Manipulation.
Revised: The lack of support for highly paralleled sampling to better cover the state space is a major impediment to the works on Deformable Object Manipulation.

Original: DaXBench fills in this gap by implementing its simulator using JAX, which support highly paralleled sampling with great automatic differentiation.
Revised: DaXBench fills in this gap by implementing its simulator using JAX, which supports highly paralleled sampling with great automatic differentiation.

Original: Using our implementation, we can finish a iteration (both forward and backward pass) forrollouts with 80 timesteps on a server with 4 2080-Ti GPUs in 3 seconds.
Revised: Using our implementation, we can finish an iteration (both forward and backward pass) for 128 rollouts with 80 timesteps on a server with 4 2080-Ti GPUs in 3 seconds.

Original: A. S AVING M EMORY
Revised: A. S AVING M EMORY

Original: Lazy Dynamic Update DaXBench optimizes the time and memory consumption of MPM by only updating the dynamic of a small region affected by the manipulation at each step.
Revised: Lazy Dynamic Update DaXBench optimizes the time and memory consumption of MPM by only updating the dynamic of a small region affected by the manipulation at each step.

Original: MPM models and updates the state and dynamics on the Particle-level; its time and memory consumption is proportional to the size of the environment, which can be arbitrarily large.
Revised: MPM models and updates the state and dynamics on the Particle-level; its time and memory consumption is proportional to the size of the environment, which can be arbitrarily large.

Original: Our insight is that, though the deformable object can potentially interact with the entire environment, only a small region will be affected at each time.
Revised: Our insight is that, though the deformable object can potentially interact with the entire environment, only a small region will be affected at each time.

Original: This inspires DaXBench to optimize the time and memory consumption of running MPM by exclusively consider the active region during the interactions.
Revised: This inspires DaXBench to optimize the time and memory consumption of running MPM by exclusively considering the active region during the interactions.

Original: Using this trick, DaXBench effectively reduces the computation time by two orders of magnitude on average.
Revised: Using this trick, DaXBench effectively reduces the computation time by two orders of magnitude on average.

Original: Taking the rope for example, the original environment is of size 128 × 128 × 128 .
Revised: Taking the rope for example, the original environment is of size 128 × 128 × 128 .

Original: However, a grid of size 32 × 6 × 32 is sufficient to cover the region of interaction between the environment and the rope.
Revised: However, a grid of size 32 × 6 × 32 is sufficient to cover the region of interaction between the environment and the rope.

Original: Therefore, we first detect the active region of size 32 × 6 × 32 , then we perform  linear transformation to shift the active region to the origin , perform the update due to deformation and interaction, finally we shift the updated active region to align with its original coordinates.
Revised: Therefore, we first detect the active region of size 32 × 6 × 32 , then we perform a linear transformation to shift the active region to the origin , perform the update due to deformation and interaction, finally we shift the updated active region to align with its original coordinates.

Original: Checkpointing method DaXBench only store the values of a few states (i.e. checkpoints), and re-evaluate the sub-step gradients online during the back-propagation to reduce the memory consumption for long horizon problems.
Revised: Checkpointing method DaXBench only stores the values of a few states (i.e. checkpoints), and re-evaluates the sub-step gradients online during the back-propagation to reduce the memory consumption for long horizon problems.

Original: The checkpointing method has been proven useful in saving the memory consumption for differentiable simulations in many prior works (Qiao et al., 2021;
Revised: The checkpointing method has been proven useful in saving the memory consumption for differentiable simulations in many prior works (Qiao et al., 2021;

Original: Chen et al., 2016; Griewank & Walther, 2000).
Revised: Chen et al., 2016; Griewank & Walther, 2000).

Original: To make the simulator end-to-end differentiable, every value in the neural network that is computed during the forward pass has to be saved, as they are needed to compute the gradient during the backward pass.
Revised: To make the simulator end-to-end differentiable, every value in the neural network that is computed during the forward pass has to be saved, as they are needed to compute the gradient during the backward pass.

Original: The computation complexity scales up quickly w.r.t.
Revised: The computation complexity scales up quickly w.r.t.

Original: the length of the task horizon.
Revised: the length of the task horizon.

Original: This is a significant problem for our simulator for high
Revised: This is a significant problem for our simulator for high

Original: DoFs deformable object manipulation, as the memory required for each timestep’s update is enormous compared to its rigid body counterpart.
Revised: DoFs deformable object manipulation, as the memory required for each timestep’s update is enormous compared to its rigid body counterpart.

Original: To overcome the large memory consumption issue,
Revised: To overcome the large memory consumption issue,

Original: DaXBench trade-offs the computation complexity slightly for large memory consumption improvement: DaXBench only stores the values of a few states (i.e. checkpoints) at the forward pass; during the backward propagation, DaXBench re-compute segments of the forward values starting from every stored state in the reverse order sequentially for the overall gradient.
Revised: DaXBench trade-offs the computation complexity slightly for large memory consumption improvement: DaXBench only stores the values of a few states (i.e. checkpoints) at the forward pass; during the backward propagation, DaXBench re-compute segments of the forward values starting from every stored state in the reverse order sequentially for the overall gradient.

Original: With this trick, DaXBench
Revised: With this trick, DaXBench

Original: [Table]
Revised: [Table]

Original: now doubles the computation complexity [Table] but saves  arbitrary amount of memory, depending on the intervals of the states saved.
Revised: now doubles the computation complexity, but saves an arbitrary amount of memory, depending on the intervals of the states saved.

Original: A. D ISCONTINUOUS G RADIENT
Revised: A. D ISCONTINUOUS G RADIENT

Original: As an end-to-end differentiable simulator
Revised: As an end-to-end differentiable simulator

Original: [Table]
Revised: [Table]

Original: the effectiveness of DaXBench rests on the accuracy of the gradient computed.
Revised: the effectiveness of DaXBench rests on the accuracy of the gradient computed.

Original: We found that for deformable object manipultation, the action not in contact with the object will cause the gradient to be discontinuous, which causes the estimate of the gradient after this action to suffer from high variance and poor accuracy in the backward propagation.
Revised: We found that for deformable object manipulation, the action not in contact with the object will cause the gradient to be discontinuous, which causes the estimate of the gradient after this action to suffer from high variance and poor accuracy in the backward propagation.

Original: To mitigate this problem, for every action input by the agent or the human user, DaXBench will perform local adjustment such that the action will always make the gripper in contact with the deformable objects.
Revised: To mitigate this problem, for every action input by the agent or the human user, DaXBench will perform local adjustment such that the action will always make the gripper in contact with the deformable objects.

Original: This ensures that our gradient to be end-to-end continuous therefore improves the accuracy of the estimated gradient.
Revised: This ensures that our gradient is end-to-end continuous therefore improving the accuracy of the estimated gradient.

Original: [Equation]
Revised: B S IM -2-R EAL G AP

Original: To verify that the dynamics of our simulated tasks has high-fidelity and  small Sim2Real gap, we carry out a real robot experiment.
Revised: To verify that the dynamics of our simulated tasks have high fidelity and a small Sim2Real gap, we carry out a real robot experiment.

Original: We deploy CEM-MPC to the PushRope task on a Kinova Gen robot.
Revised: We deploy CEM-MPC to the PushRope task on a Kinova Gen robot.

Original: CEM-MPC uses our DaX simulator as the predictive model.
Revised: CEM-MPC uses our DaX simulator as the predictive model.

Original: Given a state, CEM-MPC plans the next best push action ( x, y, z, x ′ , y ′ , z ′ ) .
Revised: Given a state, CEM-MPC plans the next best push action ( x, y, z, x ′ , y ′ , z ′ ) .

Original: The state is estimated from a point cloud image, and the Cartesian space push action is transformed to the joint space trajectory via an inverse kinematics module.
Revised: The state is estimated from a point cloud image, and the Cartesian space push action is transformed to the joint space trajectory via an inverse kinematics module.

Original: Note we are not verifying the sim2real gap caused by the inaccuracy of state estimation or kinematics; we are interested in whether the dynamics of DaX can correctly guide the CEM-MPC to succeed in the task.
Revised: Note we are not verifying the sim2real gap caused by the inaccuracy of state estimation or kinematics; we are interested in whether the dynamics of DaX can correctly guide the CEM-MPC to succeed in the task.

Original: The experiment is included in our supplementary video.
Revised: The experiment is included in our supplementary video.

Original: C RL A LGORITHMS N UMERICAL E XPERIMENT R ESULTS
Revised: C RL A LGORITHMS N UMERICAL E XPERIMENT R ESULTS

Original: We report the numerical experiment performance for the RL algorithms in Table 4.
Revised: We report the numerical experiment performance for the RL algorithms in Table 4.

Original: This is to supplement the learning curve in the main text.
Revised: This is to supplement the learning curve in the main text.

Original: This numerical results report the final performance of the learning curves in Figure 3.
Revised: These numerical results report the final performance of the learning curves in Figure [Table]

Revised: D T RANSPORTER WITH I MAGE AS I NPUT

Revised: We reported the performance of two different versions of Transporter in Table 5:

Revised: Transporter, which uses 3D particles and projects them onto an image to form a height map; and Transporter-RGB, which adds RGB channels and projects the particles onto the image plane, giving it white color with a black background.

Revised: We have found that an explicit process of rendering particles into RGB images significantly enhances the performance of Transporter.