F LUID L AB : A D IFFERENTIABLE E NVIRONMENT FOR B ENCHMARKING C OMPLEX F LUID M ANIPULATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Humans manipulate various kinds of fluids in their everyday life: creating latte art, scooping floating objects from water, rolling an ice cream cone, etc.
Using robots to augment or replace human labors in these daily settings remain as a challenging task due to the multifaceted complexities of fluids.
Previous research in robotic fluid manipulation mostly consider fluids governed by an ideal, Newtonian model in simple task settings ( e.g. , pouring water into a container).
However, the vast majority of real-world fluid systems manifest their complexities in terms of the fluid’s complex material behaviors ( e.g. , elastoplastic deformation) and multicomponent interactions ( e.g. coffee and frothed milk when making latte art), both of which were well beyond the scope of the current literature.
To evaluate robot learning algorithms on understanding and interacting with such complex fluid systems, a comprehensive virtual platform with versatile simulation capabilities and well-established tasks is needed.
In this work, we introduce FluidLab , a simulation environment with a diverse set of manipulation tasks involving complex fluid dynamics.
These tasks address interactions between solid and fluid as well as among multiple fluids.
At the heart of our platform is a fully differentiable physics simulator, FluidEngine , providing GPU-accelerated simulations and gradient calculations for various material types and their couplings, extending the scope of the existing differentiable simulation engines.
We identify several challenges for fluid manipulation learning by evaluating a set of reinforcement learning and trajectory optimization methods on our platform.
To address these challenges, we propose several domain-specific optimization schemes coupled with differentiable physics, which are empirically shown to be effective in tackling optimization problems featured by fluid system’s non-convex and non-smooth properties.
FluidLab and
FluidEngine will be publicly available.
See our project page 1 for more demos.
I NTRODUCTION
Imagine you are fishing on the lakeside.
Your hat falls into the water and starts to float out of reach.
In order to get it back, you use your hands to paddle the water gently, generating a current that pulls the the hat back into reach.
In this scenario, the water functions as a transmitting medium for momentum exchange to accomplish the manipulation task.
In fact, from creating latte art to making ice creams, humans frequently interact with different types of fluids everyday (Figure 1).
However, performing these tasks are still beyond the reach of today’s robotic systems due to three major challenges.
First, fluid systems exhibit a wide range of material models (e.g., air, liquid, granular flow, etc.
(Bridson, 2015))
that are difficult to simulate within a unified framework.
Second, describing the state of a fluid system under manipulation requires a high dimensional state space (Lin et al., 2020).
Third, the nonlinear dynamics of different coupling models pose unique challenges to efficient solution search in the state space (Schenck & Fox, 2018).
For example, although water and smoke face the same computational challenges in solving fluid equations, manipulating these two mediums in different physical contexts requires different strategies.
Previous research in robotic manipulation covering fluids mostly adopts relatively simple task settings, and usually considers tasks with a single-phase fluid, such as pouring water (Schenck & Fox, 2018; Lin et al., 2020; Sermanet et al., 2018) or scooping objects from water (Seita et al., 2022;
Antonova et al., 2022).
In this work, we aim to study the problem of complex fluid manipulation with more challenging task settings and complex goal configurations.
Since virtual environments have played an instrumental role in benchmarking and developing robot learning algorithms (Brockman et al., 2016; Kolve et al., 2017; Xiang et al., 2020), it is desirable to have a virtual platform providing a versatile set of fluid manipulation tasks.
In this work, we propose FluidLab , a new simulation environment to pave the way for robotic learning of complex fluid manipulation skills that could be useful for real-world applications.
FluidLab provides a new set of manipulation tasks that are inspired by scenarios encountered in our daily life, and consider both interactions within multiple fluids (e.g., coffee and frothed milk), as well as the coupling between fluids and solids.
Building such an environment poses a major challenge to the simulation capability: fluids encountered in our life consist of a wide spectrum of materials with different properties; in addition, simulating these tasks also requires modeling interactions with other non-fluid materials.
For example, coffee is a type of (mostly inviscid) Newtonian liquid; frothed milk behaves as viscous
Newtonian liquid (i.e., with a constant viscosity that does not change with stress)
(Figure 1, R-A), while ice cream is a type of non-Newtonian fluid (in particular, a shear-thinning fluid with its viscosity decreasing as the local stress increasing)
(Figure 1, R-B); when stirring a cup of coffee, we use a rigid stick to manipulate the coffee (Figure 1, R-A).
However, there still lacks a simulation engine in the research community that supports such a wide variety of materials and couplings between them.
Therefore, we developed our own physics engine, named FluidEngine , to support simulating the tasks proposed in FluidLab.
FluidEngine is a general-purpose physics simulator that supports modeling solid, liquid and gas, covering materials including elastic, plastic and rigid objects, Newtonian and non-Newtonian liquids, as well as smoke and air.
FluidEngine is developed using Taichi (Hu et al., 2019b), a domain-specific language embedded in Python, and supports massive parallelism on GPUs.
In addition, it is implemented in a fully differentiable manner, providing useful gradient information for downstream optimization tasks.
Closest to FluidEngine is Nvidia FleX
(Macklin et al., 2014), a physics engine that supports both rigid and deformable objects and is widely used in a variety of manipulation environments (Lin et al., 2020; Gan et al., 2020; Schenck & Fox, 2018).
However, FleX doesn’t support simulating plastic materials or air, and is not differentiable.
FluidEngine follows the OpenAI Gym API (Brockman et al., 2016) and also comes with a user-friendly
Python interface for building custom environments.
We also provide a GPU-accelerated renderer developed with OpenGL, supporting realistic image rendering in real-time.
We evaluate state-of-the-art Reinforcement Learning algorithms and trajectory optimization methods in FluidLab, and highlight major challenges with existing methods for solving the proposed tasks.
In addition, since differentiable physics has proved useful in many rigid-body and soft-body robotic tasks (Xu et al., 2021; Lin et al., 2022a; Xu et al., 2022; Li et al., 2022), it is desirable to extend it to fluid manipulation tasks.
However, supporting such a wide spectrum of materials present in FluidLab in a differentiable way is extremely challenging, and optimization using the gradients is also difficult due to the highly non-smooth optimization landscapes of the proposed tasks.
We propose several optimization schemes that are generally applicable to the context of fluids, making our simulation fully differentiable and providing useful directions to utilize gradients for downstream optimizations tasks.
We demonstrate that our proposed techniques, coupled with differentiable physics, could be effective in solving a set of challenging tasks, even in those with highly turbulent fluid (e.g., inviscid air flow).
We summarize our main contributions as follows:
• We propose
FluidLab , a set of standardized manipulation tasks to help comprehensively study complex fluid manipulation with challenging task settings and potential real-world applications.
• We present FluidEngine , the underlying physics engine of FluidLab that models various material types required in FluidLab’s tasks.
To the best of our knowledge, it is the first-of-its-kind physics engine that supports such a wide spectrum of materials and couplings between them, while also being fully differentiable .
• We benchmark existing RL and optimization algorithms, highlighting challenges for existing methods, and propose optimization schemes that make differentiable physics useful in solving a set of challenging tasks.
We hope FluidLab could benefit future research in developing better methods for solving complex fluid manipulation tasks.
R ELATED W ORK
Robotic manipulation of rigid bodies, soft bodies and fluids Robotic manipulation has been extensively studies over the past decades, while most of previous works concern tasks settings with mainly rigid objects (Zeng et al., 2020; Su´arez-Ruiz et al., 2018;
Tung et al., 2020; Chen et al., 2022).
In recent years, there has been a rising interest in manipulation problems concerning systems with deformable objects, including elastic objects such as cloth (Liang et al., 2019; Wu et al., 2019; Lin et al., 2022b), as well as plastic objects like plasticine (Huang et al., 2021).
Some of them also study manipulation problems involving fluids, but mainly consider relatively simple problems settings such as robotic pouring (Lin et al., 2020; Narasimhan et al., 2022) or scooping (Antonova et al., 2022), and mostly deal with only water (Li et al., 2018;
Schenck & Fox, 2018; Ma et al., 2018; Seita et al., 2022; Antonova et al., 2022).
In this work, we aim to study a comprehensive set of manipulation tasks involving different fluid types with various properties.
It is also worth mentioning that aside from the robotics community, a vast literature has been devoted to fluid control in computer graphics (Hu et al., 2019a; Zhu et al., 2011; Yang et al., 2013), with their focus on artistic visual effects yet neglecting the physical feasibility (e.g., arbitrary control forces can be added to deform a fluid body into a target shape).
In contrast to them, we study fluid manipulation from a realistic robotics perspective, where fluids are manipulated with an embodied robotic agent.
Simulation environments for robotic policy learning In recent years, simulation environments and standardized benchmarks have significantly facilitated research in robotics and policy learning.
However, most of them (Brockman et al., 2016; Kolve et al., 2017; Xiang et al., 2020) use either
PyBullet (Coumans & Bai, 2016) or MuJoCo (Todorov et al., 2012) as their underlying physics engines, which only supports rigid-body simulation.
Recently, environments supporting soft-body manipulation, such as SAPIEN (Xiang et al., 2020), TDW (Gan et al., 2020) and SoftGym (Lin et al., 2020) adopted Nvidia FleX
(Macklin et al., 2014) engine to simulate deformable objects and fluids.
However, FleX only supports limited types of materials, and its underlying Position-Based
Dynamics (PBD) solver has been considered not physically accurate enough to produce faithful interactions between solids and fluids.
(See Section 3 for a comparison).
Differentiable physics simulation Machine Learning with differentiable physics has gained increasing attention recently.
One line of research focuses on learning physics models with neural networks (Li et al., 2018; Pfaff et al., 2020), which are intrinsically differentiable and could be used for downstream planning and optimization.
Learning these models assumes ground-truth physics data provided by simulators, hasn’t proved to be capable of simulating a wide range of materials, and could suffer from out-of-domain generalization gap (Sanchez-Gonzalez et al., 2020).
Another approach is to implement physics simulations in a differentiable manner, usually with automatic differentiation tools (Hu et al., 2019a; de Avila Belbute-Peres et al., 2018; Xu et al., 2022; Huang et al., 2021), and the gradient information provided by these simulations has shown to be helpful in control and manipulation tasks concerning rigid and soft bodies (Xu et al., 2021; Lin et al., 2022a;
Xu et al., 2022; Li et al., 2022; Huang et al., 2021).
On a related note, Taichi (Hu et al., 2019b) and
Nvidia Warp (Macklin, 2022) are two recently developed domain-specific programming language for GPU-accelerated simulation.
FluidLab is implemented using Taichi.
F LUID E NGINE 3. S YSTEM O VERVIEW
FluidEngine is developed in Python and Taichi, and provides a set of user-friendly APIs for building simulation environments.
At a higher level, it follows the standard OpenAI Gym API and is compatible with standard reinforcement learning and optimization algorithms.
Each environment cre- ated using FluidEngine consists of five components: i) a robotic agent equipped with user-defined end-effector(s); ii) objects imported from external meshes and represented as signed distance fields (SDFs); iii) objects created either using shape primitives or external meshes, represented as particles; iv) gas fields (including a velocity field and other advected quantity fields such as smoke density and temperature) for simulating gaseous phenomena on Eulerian grids; and v) a set of user-defined geometric boundaries in support of sparse computations.
After adding these components, a complete environment is built by calling a build method, which enables inter-component communications for simulation.
(See Appendix A for a sample code of a typically environment constructed following this pipeline).
Our full source code will be released upon publication.
We encourage readers to refer to our project page for visualizations of a diverse set of scenes simulated with FluidEngine.
3. M ATERIAL M ODELS
FluidEngine supports various types of materials, mainly falling under three categories: solid, liquid and gas.
For solid materials, we support rigid, elastic and plastic objects.
For liquid, we support both
Newtonian and non-Newtonian liquids.
Both solid and liquid materials are represented as particles, simulated using the Moving Least Squares Material Point Method (MLS-MPM)
(Hu et al., 2018), which is a hybrid Lagrangian-Eulerian method that uses particles and grids to model continuum materials.
For gas such as smoke or air, we simulate them as incompressible fluid on a Cartersian grid using the advection-projection scheme (Stam, 1999).
Elastic, plastic, and liquid Both elastic and plastic materials are simulated following the original
MLS-MPM formulation (Hu et al., 2018).
For both materials, we compute the updated deformation gradient, internal force, and material stress in the particle-to-grid (P2G) step, and then enforce boundary conditions in grid operations.
Corotated constitutive models are used for both materials.
For plastic material, we additionally adopt the Box yield criterion (Stomakhin et al., 2013) to update deformations.
We refer readers to the original MPM papers (Hu et al., 2018; Stomakhin et al., 2013) for more details.
Both viscous and non-viscous liquids are also modeled using the corotated models, and we additionally reset the deformation gradients to diagonal matrices at each step.
For non-Newtonian liquids such as ice cream, we model them using the elastoplastic continuum model with its plastic component treatsd by the von Mises yield criterion (following Huang et al.
Rigid body Two types of geometric representations are used to support rigid bodies in FluidEngine.
The first type is represented by particles.
We simulate rigid objects by treating them as elastic objects first using MLS-MPM, then we compute an object-level transformation matrix and enforce the rigidity constraint.
Another type of rigid object is represented using signed distance fields (SDFs), usually generated by importing external meshes.
This representation is more computation-efficient since it computes its dynamics as a whole, generally used for meshes with big sizes.
Gas We develop a grid-based incompressible fluid simulator following (Stam, 1999) to model gaseous phenomena such as air and smoke.
A particle-based method is not well suited in these cases due to its (weakly) compressible limit.
Inter-material coupling MPM naturally supports coupling between different materials and particles, computed in the grid operations.
For interactions (collision contact) between mesh-based objects and particle-based materials, we represent meshes as time-varying SDFs and model the contact by computing surface normals of SDFs and applying Coulomb friction (Stomakhin et al., 2013).
For interaction between mesh-based objects and the gas field, we treat grid cells occupied by meshes as the boundaries of the gas field and impose boundary conditions in the grid operations.
Coupling between gas and particle-based materials is also implemented at the grid level, by treating grid cells occupied by particles as boundaries.
3. D IFFERENTIABILITY AND R ENDERING
FluidEngine is implemented based on Taichi’s auto diff system to compute gradients for simple operations.
We further implement our own gradient computations for complex operations such as
SVD and the projection step in the gas simulation, and efficient gradient-checkpointing to allow gradient flow over unlimited time horizons, unconstrained by GPU memory size.
FludiEngine also comes with a real-time OpenGL-based renderer.
More details are in Appendix B.
[Table]
[Table]
3. C OMPARISON WITH OTHER SIMULATION ENVIRONMENTS
There exist many simulation engines and environments for robotics research, and here we compare
FluidLab and FluidEngine with some popular ones among them, including PyBullet (Coumans &
Bai, 2016), MuJoCo (Todorov et al., 2012), FleX (Macklin et al., 2014), SoftGym (Lin et al., 2020),
PlasticineLab (Huang et al., 2021), SAPIEN (Xiang et al., 2020), TDW (Gan et al., 2020), DiffSim (Qiao et al., 2020), DiSECt (Heiden et al., 2021), PhiFlow (Holl et al., 2020), JAX-Fluids (Bezgin et al., 2023) and DEDO (Antonova et al., 2021).
Table 1 shows a comparison in terms of differentiability and supported material types.
FluidLab covers a wider range of materials compared to existing environments.
In addition, many of these environments rely on physics engines written in C++, and some are not open-sourced (such as FleX), making interpretability and extensibility a major hurdle, while FluidLab is developed in Python and Taichi, enjoying both GPU-acceleration and user-extensibility. In order to validate the accuracy and reliability of FluidEngine, we conducted a number of validation simulations. See Appendix F for more details.
Moreover, in order to place
FluidEngine and FluidLab more clearly in the literature, we present a list of detailed discussions with relevant differentiable simulators in Appendix G.
F LUID L AB M ANIPULATION T ASKS
FluidLab presents a collection of complex fluid manipulation tasks with different fluid materials and their interactions with other objects.
In general, we observe that manipulations tasks involving fluids can be categorized into two main groups: i) where fluid functions as a transmitting media and is used to manipulate other objects which are otherwise unreachable, and ii) where fluid is the manipulation target and the goal is to move or deform the fluid volume to a desired configuration.
We present a suite of 10 different tasks in FluidLab, as shown in Figure 2, where in each task an agent needs to manipulate or interact with the fluids in the scene to reach a specific goal configuration, falling under the aforementioned two categories.
4. D ETAILS OF E VALUATION T ASKS
FluidLab contains 10 different manipulation tasks presented in Figure 2, all of which are motivated by real-world scenarios where an agent interacts with certain types of fluids. We additionally discuss the our reasoning behind the choices of our task selections in Appendix H.
Fined Grained Pouring A glass filled with two types of non-Newtonian liquid with different densities, colored as blue and white respectively. The task is to pour the light blue liquid out of the glass while retaining as much white heavy liquid as possible. Gathering (Easy & Hard) An agent equipped with a rigid plate is able to paddle the water in a tank, but its motion is restricted within the green area.
The goal is to make water currents so that the two floating objects will be gathered towards the marked goal location.
The easy has a relatively space while in the hard setting, the spatial configuration is more complex and the two objects need to move in different directions.
Transporting A water jet bot is required to move an object of interest to a randomly sampled goal location.
The agent is only allowed to move within the green area, and thus can only transport the object using the water it injects.
Mixing Given a cube of sugar put in a cup of coffee, the agent needs to plan an sequence of stirring action and mixes the sugar particles with the coffee in the shortest period of time. Latte Art (Pouring & Stirring) We consider two types of latte art making tasks: the first one is making latte art via pouring, where an agent pours milk into a cup of coffee to form a certain pattern; the second one is via stirring, where a layer of viscous frothed milk covers a cup of coffee, and the agent needs to manipulate a rigid stick to stir in the cup to reach a specified goal pattern.
Note that due to the nonlinear fluid dynamics governing the system, these two tasks are much more complex than simple drawing on a static canvas.
For example, during pouring, where a milk particle ends up being at the end of the task may be far away from where it was originally poured from, due to continuous interaction with the coffee beneath it and motion of the fluid body caused by pouring.
Ice Cream (Static & Dynamic)
This task involves policy learning concerning a non-Newtonian fluid – ice cream.
We consider two settings here.
The first one is a relatively simple and static setting, where we control an ice cream emitter that moves freely in space and emits ice cream into a cone and form a desired spiral shape.
The second setting is more dynamic, where ice cream is squeezed out from a machine and an agent controls the biscuit cone.
The goal configuration is defined with respect to the cone itself.
During the process, the agent needs to reason about the dynamics of the icecream and its interaction with the cone since the ice cream moves together with the cone due to friction.
Air Circulation
The agent controls the orientation of an air conditioner (denoted in green in the figure) which generates cool air (colored in blue) in a house with multiple rooms, initially filled with warm air (colored in orange), and the goal is to cool down only the upper left and the right room, while ensuring the lower left room maintains at its original temperature.
Completing this challenging task requires fine-grained control and reasoning over the motion of the cooling air.
4. T ASK AND A CTION REPRESENTATIONS
We model each task as an finite-horizon Markov Decision Process (MDP).
An MDP consists of a state space S , an action space A , a transition function T : S ×
A → S , and a reward function associated with each transition step R : S×A×S → R .
The transition function is determined by the physics simulation in FluidLab.
We discuss the loss and reward for each task used for experiments in Section 5.
The goal of the agent in the environment is to find a policy π ( a | s ) that produces an action given the current state, and maximize the expected total return E π [Σ Tt =
γ t R ( s t , a t )] , where γ ∈ (0 , 1) denotes the discount factor and T is the horizon of the environment.
State, Observation, and Action Space We represent MPM-based materials with a particle-based representation, using a N p × d p state matrix, where N p is the number of all particles in the system, and d p = 6 contains the position and velocity information of each particle.
To represent the gas field, which is simulated using a grid-based representation, we use another N c × d c matrix to store information of all grid cells, where N c is the number of cells in the grid, and d c is a task-dependent vector.
In the air circulation task, d c = 4 contains the velocity (a 3-vector) and temperature (a scalar) of each cell.
In addition, the state also includes the 6D pose and velocity (translational and rotational) information of the end-effector of the robotic agent in the scene.
All the tasks use an MPM grid of size 64 3 , with around 100,000 particles, and a 128 3 grid for gas simulation.
The full action space of FluidLab is a 6-vector, containing both translational and angular velocity of the agent’s end-effector.
For more details of our task settings, their simulations, and the input to RL policies, please refer to Appendix D.1.
E XPERIMENTS
We evaluate model-free RL algorithms, sampling-based optimization methods as well as trajectory optimization using differentiable physics coupled with our proposed optimization techniques in FluidLab.
We discuss our optimization techniques, and result findings in this Section.
For detailed loss and reward design, please refer to Appendix D.2.
5. T ECHNIQUES AND O PTIMIZATION S CHEMES U SING D IFFERENTIABLE P HYSICS
Differentiable simulation provides useful gradient information that can be leveraged for policy optimization.
However, oftentimes the dynamics function could both be highly non-smooth (e.g. in case of a contact collision, where the gradient is hard to obtain), and exhibit a complex optimization landscape where the optimizer can easily get trapped in local minima.
We discuss a few techniques and optimization schemes used in our experiments here for obtaining more informative gradients and leveraging them more effectively in the context of fluid manipulation.
Soft Contact Model
In our simulation, the contact between particle-based fluid bodies and SDFbased rigid objects occurs in the grid operations.
Such contact induces drastic state changes during the classical MPM process.
We adopt a softened contact model in our simulation: we consider contact occurs when the particles and the rigid objects are within a certain distance threshold ϵ , and compute an after-collision velocity v c .
Afterwards, we blend this velocity with the original velocity of the objects using a distance-based blending factor α , given by v new =
αv c
+ (1 − α ) v original , where α = min { exp( − d ) , 1) } and d is the distance between the two objects.
This provides a smoothed contact zone and thus offers smoothed gradient information during contact.
Temporally Expanding Optimization Region During gradient-based trajectory optimization, if we consider the full loss computed using the whole trajectory with respect to a certain goal configuration, the whole optimization landscape could be highly nonconvex and results in unstable optimization.
In our experiments, we warm start our optimization with a restricted temporal region of the trajectory and keeps expanding this region steadily whenever a stabilized optimization spot is reached for the current region.
This greatly helps stabilize the optimization process using differentiable physics and produces better empirical results, as we show in Section 5.2.
Gradient Sharing within Fluid Bodies Many optimization scenarios involving fluids could be highly non-convex.
Consider a simple pouring task where the goal is to pour all the water from a mug, with the loss defined as the distance between the ground floor and all the water particles.
During the exploration phase, if a water particle was never poured out of the mug, it is trapped in a local minimum and the loss gradient won’t be able to guide towards a successful pouring.
Thanks to the continuity of fluid materials, we are able to share gradients between adjacent particles in a fluid body and use those informative gradients to guide other particles.
In our experiments where fluid is required to reach a goal configuration, we enable gradient sharing by updating the gradient on each particle via accumulating neighboring particle gradients as follows:
∂ L ∂q
[Equation]
d
[Equation] where j iterates over all neighboring particles of i , q represents particle properties such as velocity or position, and w is a kernel function conditioned on both the distance between i and j and the loss of particle j , assigning a higher weight to neighboring particles who incurred a lower loss at the previous iteration. [Table]
5. M ETHOD E VALUATION WITH F LUID L AB T ASKS
We evaluate our proposed optimization schemes coupled with differentiable physics (DP), modelfree RL algorithms including Soft Actor-Critic (SAC) (Haarnoja et al., 2018) and Proximal Policy
Optimization (PPO) (Schulman et al., 2017), CMA-ES (Hansen & Ostermeier, 2001), a samplingbased trajectory optimization method, as well as PODS (Mora et al., 2021) an method combines RL and differentiable simulation.
We use the same accumulated reward to evaluate all the methods.
Additionally, we include the performance of an oracle policy controlled by a human operator as a reference.
The reward learning curves are shown in Appendix D.2, and we report the final performance of all methods upon convergence in Table 2.
Readers are encouraged to refer to our project page for more qualitative results.
Model-free RL Algorithms RL algorithms demonstrate reasonable learning behaviors in tasks with relatively small observation spaces, such as Transporting (where only a subset of water particles are present in the scene at each step) and Air Circulation.
However, in environments such as Latte
Art, Mixing, and Ice Cream, RL methods struggle to generate a reasonable policy.
We suspect that this is because 1) the high-dimensional space used to describe the fluid system, which is hard for RL algorithms to abstract away useful information about the system within limited exploration iterations and training budget, making it less sample efficient than DP, and 2) the non-linear dynamics of the complex fluids such as viscous frothed milk and the non-Newtonian ice creams, as well as their interactions with other solid materials, make policy search of RL policies based on sampled actions extremely difficult since randomly sampled actions and rewards are not informative enough in conveying contact and deformation information about the fluid systems.
Sampling-based trajectory optimization CMA-ES does reasonably well in many tasks with simple spatial configuration as goals.
However, it struggles in fine-grained shape matching tasks ( e.g. Ice
Cream) and fine-grained control tasks ( e.g. pouring).
This is because CMA-ES searches for action sequences via trajectory sampling, and thus cannot handle delicate interactions with fluid materials well.
Differentiable physics-based trajectory optimization In most of the tasks, the gradients provided by our differentiable simulation are able to guide trajectory optimization towards a good optimization spot, resulting in near-oracle performance.
It also demonstrates better sample-efficiency than other methods in most tasks.
We additionally compare with a hard version of differentiable physics (DP-H), which uses the gradient to optimize trajectory, but without using the optimization techniques proposed in Section 5.1.
DP-H does similarly well on tasks with relatively simple settings, but fails to produce good trajectories in scenarios with interactions between fluids and other materials bodies, as well as in tasks with more complex and non-convex optimization landscapes, such as Pouring with fine-grained control objective.
The experimental results demonstrate that the gradient information provided by our differentiable simulator, coupled with the optimization schemes we proposed, provides informative optimization information when dealing with complex fluid systems. We additionally evaluate the optimized trajectories in a real-world robotic setup and show that the learned trajectories can perform the desired tasks reasonably well in the real world. See Appendix
E for more details.
Comparison with PODS PODS is a method that combines differentiable simulation and RL.
It performs well in relatively convex tasks such as Latte Art (Pouring), and even outperforms DP and
DP-H on simple tasks like transporting, mainly due to its exploratory nature from RL. However, it is not able to match performance in other complex tasks, especially in those where DP shows an clear advantage over DP-H and RL methods. We believe this is because PODS only uses local gradient information without our proposed optimization techniques; also, it is additionally learning a closedloop policy which maps observation to action, where DP is only optimizing a trajectory. In fact, what PODS is learning is precisely the next step of our research, where a better scene representation to describe the fluid system is needed to better distill DP-optimized trajectory to a closed-loop policy network.
This comparison suggests that a future research direction is how to encode the scene in a more informative representation for better policy learning, such as a pointcloud based representation; in addition, we expect better performance by combining our proposed optimization landscapes with methods that use both differentiable simulation and RL.
These remain as our future work.
Potential future research directions
In our experiments above, we evaluated a range of policy learning and trajectory optimization methods.
Our results show that when coupled with our proposed optimization scheme, differentiable physics can provide useful gradient information to guide trajectory optimization, and show reasonable performance in a range of complex task settings, even in those with fine-grained shape matching goals and highly unstable fluid systems such as air flow.
Meanwhile, our results suggested a number of major challenges in solving complex fluid manipulation methods.
First of all, the model free RL methods and also the method combining RL and differentiable simulation for policy learning struggle to produce competitive results.
We believe one main reason for this is it’s difficult for the policy to informatively interpret the high-dimensional state of the complex fluid systems.
It’s worth investigating what would be an efficient representation for learning such closed-loop policies: point cloud-based representation or image input could be a good starting point, and more abstract representations with semantic understanding of the material properties and appearances are also worth studying.
Second, one clear goal is to combine our optimization scheme with policy learning, and to distill optimized trajectories using differentiable physics into more general close-loop control policies.
Thirdly, although our current task selections cover a range of different scenarios and materials, they are still relatively short-horizon, in the sense that there’s usually only a single action mode.
It would be very interesting to extend task design to more realistic latte-art making, which requires action switching between both pouring and stirring.
One promising direction is to learn a more abstract dynamics model of the fluid system and use it for long-horizon action planning and skill selection.
Also, our current latte art tasks rely on dense reward signal computed from a temporal trajectory of goal patterns, and a more abstract and learned dynamics model would probably allow optimization using a static goal pattern.
Last but not least, our real world experiments show that there’s still a perceivable gap between our simulated material behavior and the real world environments.
Improving both simulation performance and close-loop policy learning are needed to enable robots to perform better in more complex tasks, such as more realistic latte art making.
We presented FluidLab, a virtual environment that covers a versatile collection of complex fluid manipulation tasks inspired by real-world scenarios.
We also introduced FluidEngine, a backend physics engine that can support simulating a wide range of materials with distinct properties, and also being fully differentiable.
The proposed tasks enabled us to study manipulation problems with fluids comprehensively, and shed light on challenges of existing methods, and the potential of utilizing differentiable physics as an effective tool for trajectory optimization in the context of complex fluid manipulation.
One future direction is to extend towards more realistic problem setups using vision as input.
One potential way is to distill policy optimized using differentiable physics into neural-network based policies (Lin et al., 2022a), or use gradients provided by differentiable simulation to guide policy learning (Xu et al., 2022).
We also plan to test policies trained using our simulation in real-world setups.
Furthermore, since MPM was originally proposed, researchers have been using it to simulate various realistic materials with intricate properties, such as snow, sand, mud, and granular materials.
These more sophisticated materials can be seamless integrated into the MPM simulation pipeline in FluidLab, and will be helpful for studying manipulation tasks dealing with more diverse set of materials.
We believe the standardized task suites and the general purpose simulation capability of FluidLab will benefit future research in robotic manipulation involving fluids and other materials.
B D ETAILS ON F LUID L AB ’ S D IFFERENTIBILITY AND R ENDERING
Differentiability and gradient checkpointing We use the automatic differentiation tool (autodiff) provided by Taichi to get gradients for most of our computations; for more complex computations and kernels not supported by Taichi’s autodiff, we implemeneted gradient computation ourselves, including SVD computations and the projection step in gas simulation.
One additional challenge for making the whole simulation fully differentiable is enabling gradient flow through time, which requires bookkeeping system states of all timesteps.
Since most of FluidLab’s tasks have tens of thousands of simulation steps, it is impossible to fit the whole computation graph into the GPU memory.
In order to address this, we implemented efficient gradient checkpointing to enable gradient flow over unlimited time horizons.
During the forward process of the simulation, only a local trajectory of the environment states are stored in the GPU memory for fast computation.
Whenever the GPU memory is full, we store the latest state of the environment to a caching device (either CPU memory or local disks), and clear up the GPU memory for further simulation.
Once a complete forward pass is done, we restore the cached states and run forward simulation again to fulfill the GPU memory with the local chunk of simulation trajectory.
Note that instead of caching all intermediate states, we only store one single step of the states at every caching step, and run simulation again to restore the states of the local trajectory; therefore, the i/o step of such caching operations is very efficient.
This enables us to propagate loss and gradient information backward in time through a simulation trajectory with arbitrary number of steps, unconstrained by the GPU memory size.
Rendering We implemented a photo-realistic renderer for visualizing scenes created with FluidEngine.
The renderer is developed in C++ using OpenGL, supports GPU-accelerated real-time rendering.
Part of our rendering pipeline was modified based on FleX’s rendering system, enhanced with various features such as particle-level colorization, headless rendering, dynamic object loading, smoke rendering, etc.
We also provided a set of python APIs to dynamically update the scene configuration from within Python.
The renderer supports rendering materials as either particles or fluids.
Particle-based rendering is more interpretable for visualization and debugging purposes, while fluid-based rendering is more photo-realistic and allows possible image-based sim-to-real transfer applications in the future. We present a comparison of these two rendering methods in Figure 3. C R EWARD L EARNING C URVE FOR THE E VALUATED M ETHODS
Here we include the reward learning curves of our evaluated methods in Figure 4.
D F LUID L AB T ASK AND E VALUATION D ETAILS D. T ASK D ETAILS
For RL algorithms, we down sample the number of particles and cells for each fluid body in the scene to avoid a intractably huge observation space.
The state matrix is then flattened and fed into an MLP policy network.
We do not evaluate image-based RL methods as many tasks FluidLab are 3D and occlusion could be a potential problem, and leave such evaluations for future work.
All our tasks use a simulation step of 2e-3 seconds, each containing 10 substeps of 1e-4 seconds for ensuring simulation stability.
All our proposed tasks contain around 100,000 particles and runs in real time (60 FPS where each frame is a simulation step) on a desktop computer equipped with an Nvidia RTX 3080 GPU and a Intel i7-8700K CPU, and the typical GPU usage is under 50%. Each task has a task-specific action range and action space, with certain dimensions of the full action-space locked. We present our task-specific settings for action spaces, observed number of particles per fluid body and observed number of grid cells in Table 3.
Detailed properties of all materials used are listed in Table 4.
D. L OSS AND R EWARD
Latte Art (Pouring) , Latte Art (Stirring) , Ice Cream (Static) and Ice Cream (Dynamic) : These tasks have a set of predefined goal patterns, and the loss is defined by summing up the Chamfer [Table]
Trans.
Velocity
Rot.
Velocity per fluid body for gas field
Latte Art (Pouring)
Along x and z - [Table]
Latte Art (Stirring)
Along x and z - [Table]
Ice Cream (Static)
Along x, y and z [Table]
Ice Cream (Dynamic)
Along x, y and z [Table]
Transporting
Along x and z
Around y [Table] Mixing
Along x and z [Table] Pouring
[Table] Around z [Table] Gathering (Easy)
Along x and z -
[Table] Gathering (Hard)
Along x and z [Table]
Air Circulation
[Table] [Table]
Light Liquid in Pouring
[Table]
Heavy Liquid in Pouring
[Table]
Object in Transporting
[Table]
distance between particles in the current state of the scene and the particles in the goal pattern at each time step.
Ideally, the goal pattern should be a static goal shape of the final step, but using a single goal pattern would be extremely challenging for RL methods: if only the last step of the trajectory is used for reward computation, such an extremely sparse reward will make learning infeasible; computing a dense reward based on the distance between the current state of each timestep and the goal pattern will also be non-informative: due to the dynamic movement within the fluid bodies, mimicing the static goal pattern at each timestep will not result in successful learning.
Therefore, in order to make an informative comparison, we use a temporal trajectory of the desired goal pattern to compute the loss L = Σ t CD( s t , s goalt ) , where CD denotes Chamfer Distance.
Transporting , Pouring , Gathering (Easy) and Gathering (Hard) :
These tasks use a single spatial location as the goal and the loss is computed as: L =
Σ i ∥ p i − p goal ∥ , which is summed over all particles of the object(s) to be manipulated.
For Pouring , we apply this loss between the particles of the upper layer fluid and the ground location, with an additional loss attracting the particles of the lower layer fluid to their initial positions.
Mixing The goal of this task is to mix the sugar particles with the coffee.
The loss is designed to maximize the spatial coverage of the sugar particles and is defined as L = − Σ i Σ j ∥ p
i − p j ∥ , where both i and j iterate over all particles originally belonging to the sugar cube.
Air circulation There are 27 sensors placed in the house, with 9 sensors in each room uniformly.
The loss is defined as L = Σ ul | T ul − T cool | + Σ r | T r − T cool | + Σ ll | T ll − T warm | , where T represents temperature, and ul, r and ll denote sensors in the upper left, right and lower left room respectively.
We use the losses discussed above for optimization-based methods; for model-free RL methods, the reward of each task is simply defined as R =
c 1 − c 2 L , where c 1 and c 2 are constants to ensure a reasonable reward scaling for each task.
E S IM - TO - REAL T RANSFER
Our simulation platform provides an efficient test bed and useful gradient information for developing algorithms to deal with complex fluid manipulation problems, testing how the optimized policies perform in a real-world setting would be very helpful and informative in evaluating the sim-to-real gap and suggesting future research directions.
Therefore, we conducted experiments on a real robot system, with a 7-DoF Franka Emika robot equipped with a parallel jaw gripper.
We picked four representative tasks proposed in FluidLab, including Latte Art (Stirring), Ice Cream (Dynamic),
Gathering and Mixing, and set up corresponding real world scenarios.
We optimize the trajectories in simulation, and then command the robot to execute them using velocity control in the corresponding real-world scenarios.
(See our project site for qualitative results.)
It can be observed that although there’s certain sim-to-real gap in terms of material behavior and dynamics, when we apply the simulation-optimized trajectories to real-world, the tasks can be completed to a reasonable extent.
The sim-to-real gap mainly comes from the simulation inaccuracy and difference in material properties between sim and real.
For example, in the latte art experiment, the frothed milk behaves a bit differently than in simulation: it’s more sticky and tend to mix again after the latte art pen passes.
This could be further improved in simulation by developing more accurate material models, as well as increasing the simulation resolution and number of particles, which we will strive to improve in our future work.
Another example is the ice cream experiment, where it is difficult to maintain a steady flow speed of the ice cream, resulting in slight different end shape of the ice cream compared to the simulated one.
Although there exists such sim-to-real gap in material dynamics, the robot is able to complete the proposed tasks reasonably well.
We would also like to acknowledge that given our current resources, we are not able to conduct all the proposed tasks in real world, such as the air circulation task, which remains as our future work.
However, We believe the current 4 selected tasks cover a representative range of materials and task settings, and such experiments could shed light on the value of our simulator as a test bed for robotic research, point out potential sim-to-real gap and suggest possible improvements in our future work.
F V ALIDATION E XPERIMENTS FOR F LUID E NGINE
One useful way of validating the correctness of simulation engines is to run simulation experiments and verify if the produced simulation results match known physical phenomena.
Below we list a number of classical physical phenomena that are typically observable in rigid body and computational fluid dynamics.
We testified our simulation engine by running simulations to see if it can successfully reproduce the listed behaviors.
Visual results of such simulations are included in our project website.
• K´arm´an vortex street: When a flow of fluid is separated by blunt bodies, a repeating pattern of swirling vortices would appear due to vortex shedding.
We simulate a flow of fluid flowing through a rigid cylinder and visualize the top-down view.
We colorized the fluid around the cylinder, and it can be seen that our simulator can faithfully produce a street of vortices.
• Magnus effect: This is an observable phenomenon commonly associated with a spinning object moving through a fluid.
The path of the spinning object is deflected in a manner not present when the object is not spinning.
One example of such phenomenon seen in daily life is the screw shot in football playing
This is an observable phenomenon commonly associated with a spinning object moving through a fluid.
The path of the spinning object is deflected in a manner not present when
the object is not spinning.
One example of such phenomenon seen in daily life is the screw shot in football playing.
Here we simulate a spinning ball in a fluid body.
It can be observed that the spinning motion produces a deflection in the motion trajectory of the ball, and the extent of the deflection is dependent on its spinning velocity.
• Buoyancy: We verify here if objects with different density in a liquid body would incur different magnitude of buoyancy.
It can be observed that the red all would float on the water surface due to buoyancy.
• Incompressibility and stable volume: Liquids are generally considered incompressible.
Our MPM-based engine simulates weakly-compressible liquids, where the momentum and mass are updated every step via consecutive grid-particle-grid information passes.
Therefore, one concern is if the simulated fluid bodies can maintain consistent pressure and volume through long-horizon simulations, without momentum and pressure loss.
We verified our simulator can simulate a stable fluid body that maintains steady volume and pressure after 10,000 simulation steps.
• Conservation of momentum: When objects are in collision, their motions obey conservation of momentum.
Here we test the collision of two objects with exactly the same mass on a frictionless floor, where it can be observed that they exchanged their momentum after the collision as expected.
• Varying bouncing behaviors with varying plasticity: Here we show different behaviors of the object bouncing on the floor when we vary the plasticity of the object.
From left to right, we increase the plasticity of the object.
Our simulator can simulate bouncing behavior with varying plasticity, where it can be observed energy dissipates faster with a greater plasticity.
• Rayleigh–Taylor instability: This phenomenon describes the instability of an interface between two fluids with different densities, which occurs when the heavy component is above and pushing the light one.
Here we simulate the behavior of two layers of fluids with different densities, where it can be observed that a plume of heavy fluid with small vortices emerges and evolves due to the interfacial instabilities.
• Dam break: This is a classical engineering test case, where a fluid volume falls due to gravity and splash within the domain.
We successfully reproduce this phenomenon with our simulation engine.
Our simulators successfully reproduced these physical phenomena, as we show on our project site.
We believe running such validation experiments could informatively help demonstrate the accuracy and reliability of our simulation engine.
G C OMPARISON WITH OTHER DIFFERENTIABLE SIMULATORS G. SPN ETS
SPNets is a pioneering work in robotic manipulation of fluids.
It is essentially a fluid simulator implemented as neural network layers, which makes it differentiable.
Therefore, it is more like a differentiable simulator itself rather than a policy learning method.
The parameters of SPNets are the physical parameters of the fluids.
One limitation of SPNets is that its simulation capability: it only supports simulating single-phase non-Newtonian fluids, and only supports one-way coupling from static boundaries to the fluids i.e. it doesn’t support computing effects of fluids on other objects.
In this work, one of our key insights is that in order to study realistic fluid manipulation problems that are motivated by real-world scenarios with realistic complexity, it’s crucial to be able to support a wide range of materials and their inter-coupling.
Our motivation is to study problems beyond the single-phase water tasks proposed in SPNets, such as single-phase water pouring and catching, where only the effect of manipulators on the water is considered.
Therefore, on the simulator side,
SPNets is not able to simulate most of our propose tasks.
In addition, on the task solving side, in the
SPNets paper the author did propose to use gradients provided by the differentiable simulation to optimize trajectories for liquid control.
Which is similar to our differentible physics based optimization method (DP-H), except that we proposed a few optimization techniques (DP) to help solving the tasks.
G. D IFF S IM
DiffSim is a differentiable simulation platform that leverages mesh-based representation and implicit differentiation, which bring faster computation and less memory cost.
However, mesh-based representation is not able to simulate highly deformable materials such as fluids.
In contrast to them, we opt for a particle-based representation since our work focuses on simulating fluid systems and relevant manipulation skills.
We use MPM-based method which naturally supports particle-based materials ranging from solid to fluids, and also the coupling between them.
G. D I SEC T
DiSECt is a differentiable tool tailored towards robotic cutting of soft materials.
It also opts for a mesh-based representation and FEM to simulate dynamics, which is ideal for computing physical phenomena in cutting scenarios, such as plastic deformation, fracture, and crack propagation.
Our work differs from them in that FluidEngine is a general-purpose simulator, and our focus is on manipulation scenarios involving different types of fluids.
Therefore, although our simulator does not specialize in special cutting scenarios, it can support a wide range of daily manipulation tasks involving interaction between various material types, ranging from solid, liquid to gas.
G. T AICHI AND D IFF T AICHI
Taichi is a domain-specific programming language that provides GPU-accelerated parallel computation, and we use it to implement FluidEngine and FluidLab.
DiffTaichi is the automatic differentiation system of Taichi that supports gradient computation of numerical operations coded using
Taichi.
On the other hand, Taichi and DiffTaichi come with many example codes of simulating different scenarios, but they are scene-specific simulation examples and there’s a big gap between them and an actual simulation environment that can be used for repeatable robotics research.
We list a number of major difference here.
1) DiffTaichi doesn’t support building computation graph and gradient checkpointing, which is necessary for computing and storing gradient flow over longhorizon manipulation tasks, especially when GPU memory is a major bottleneck.
Therefore, unlike in PyTorch, any computation step involving gradient flow needs manually setting up the computation graph, and our framework handles that. 2) Taichi does provide MPM-based simulation examples, but they don’t support functions such as mesh loading, SDF computation, SDF-based collision detection, friction modelling, mesh to particle conversion, etc. 3)
Taichi doesn’t support grid-based 3D smoke simulation, and doesn’t support coupling between smoke, liquid, and mesh-based rigid objects, which is precisely one of the contributions of our platform, since such coupling is necessary for studying realistic fluid manipulation problem.
4) DiffTaichi only provides gradient computation of simple operations, not supporting gradients for e.g. SVD computations and the projection step in gas field simulation. 5)
FluidEngine provides a unified configuration system and interface for building custom environments and specifying robotic agents, and also provides function for storing and accessing gradients of the intermediate states and robot actions.
The examples provided with
Taichi typically requires saving scene configurations to a file and using an external rendering software such as Houdini for visualization, while we implement a stand alone OpenGL-based renderer that runs in real time and provides easy visualization and also capability for developing image-based policy learning method.
7) FluidLab is a set of standardized manipulation tasks for developing and evaluating robot learning algorithms, built on top of FluidEngine.
This is similar to SoftGym versus FleX. In summary, FluidEngine is a piece of complete software with various APIs that can be used easily for building up robotic research environments, just like what people are able to do with
PyBullet or Mujoco, except that FluidEngine additionally supports deformable and fluid materials, while also providing gradient information.
G. P HI F LOW
PhiFlow is a differentiable simulation toolbox.
However, the major functionality is a underlying differentiable PDE solver, which can be used to protype small scenes with fluids but is not capable of building up realisitc 3D robotic manipulation scenarios.
It requires manually specifying boundary conditions for fluid simulation, which is not realistic if we want to use actual object meshes for
Materialtype
Contactwithmanipulator
Roleoffluids
Keychallenge/uniquecharacteristic
LatteArt(Pouring)
Inviscidandviscousliquid
Indirect
Manipulationtarget
Couplingbetweendifferentliquids LatteArt(Stirring)
Inviscidandviscousliquid
Direct
Manipulationtarget
Couplingbetweenliquidsandtherigidmanipulator IceCream(Static)
Non-Newtonionfluid
Indirect
Manipulationtarget
Finegrainedshapematchingofnon-Newtonianfluid IceCream(Dynamic)
Non-Newtonionfluid
Direct
Manipulationtarget
Additionalinteractionwiththerigidmanipulator Transporting
Inviscidliquidandrigidsolid
Indirect
Transmittingmedium
Useliquidtoindirectlymanipulatedistantobjects Mixing
Inviscidandviscousliquid
Direct
Manipulationtarget
Couplingbetweendifferentliquidbody Pouring
Inviscidliquid
Direct
Manipulationtarget
Fine-grainedgoalconfiguration Gathering(Easy) liquidandelasticsolid
Direct
Transmittingmedium
Useliquidtoindirectlymanipulatedistantobjects Gathering(Hard) liquidandelasticsolid
Direct
Transmittingmedium
Morecomplexenvironmentconfiguration AirCirculation
Gas
Indirect
Manipulationtarget
Involvingdynamicsofhighlyunstableairflow manipulation.
Also, it relies on grid-based simulation for fluids, while our particle based simulation is more ideal for simulating liquids, which does not require boundary specifying and supports sparse computation.
Moreover, PhiFlow is a special-purpose simulator specific to flow simulation, and doesn’t support other material types such as solids, which is important for conducting robotics research in manipulation.
G. JAX-F LUIDS
Similar to PhiFlow, JAX-Fluids is a special-purpose differentiable solver tailored to flow simulation. Therefore, it shares the similar limitations of PhiFlow as discussed above: it’s not designed for general purpose multi-material simulation, not supporting dynamics of other materials such as rigid and plastic objects, and does not support features such as mesh processing, scene building, and realistic rendering. In contrast, FluidEngine is a full-stack simulation platform providing such features and useful for conducting robotic manipulation research. H D ISCUSSION ON TASK SELECTIONS
When designing our tasks, we would like them to cover a wide range of variations, and these are the main factors that we consider: 1) varying material type and their coupling, from inviscid liquid, to viscous liquid, to non-newtonian fluid, to gaseous fluid, 2) the contact between the robot manipulator and the fluid body (direct contact or indirect emitting), and 3) the role of fluids, as a transmitting medium or the manipulation target. Therefore, we came up with the 10 selected tasks. In Table 5, we compare the selected tasks over the above 3 factors, and identify the unique characteristics of each task. We believe our proposed suite of task selections covers a wide range of fluid manipulation tasks encountered in our daily life, and should be useful as a set of standardized tasks for studying fluid manipulation problem.
[Table]
In Table 6, we list the optimization techniques we found useful for each task.
DP and DP-H are the same for tasks not listed here.