  We present a differentiable soft-body physics simulator that can be composed
with neural networks as a differentiable layer. In contrast to other
differentiable physics approaches that use explicit forward models to define
state transitions, we focus on implicit state transitions defined via function
minimization. Implicit state transitions appear in implicit numerical
integration methods, which offer the benefits of large time steps and excellent
numerical stability, but require a special treatment to achieve
differentiability due to the absence of an explicit differentiable forward
pass. In contrast to other implicit differentiation approaches that require
explicit formulas for the force function and the force Jacobian matrix, we
present an energy-based approach that allows us to compute these derivatives
automatically and in a matrix-free fashion via reverse-mode automatic
differentiation. This allows for more flexibility and productivity when
defining physical models and is particularly important in the context of neural
network training, which often relies on reverse-mode automatic differentiation
(backpropagation). We demonstrate the effectiveness of our differentiable
simulator in policy optimization for locomotion tasks and show that it achieves
better sample efficiency than model-free reinforcement learning.
