  We consider the problem of estimating an object's physical properties such as
mass, friction, and elasticity directly from video sequences. Such a system
identification problem is fundamentally ill-posed due to the loss of
information during image formation. Current solutions require precise 3D labels
which are labor-intensive to gather, and infeasible to create for many systems
such as deformable solids or cloth. We present gradSim, a framework that
overcomes the dependence on 3D supervision by leveraging differentiable
multiphysics simulation and differentiable rendering to jointly model the
evolution of scene dynamics and image formation. This novel combination enables
backpropagation from pixels in a video sequence through to the underlying
physical attributes that generated them. Moreover, our unified computation
graph -- spanning from the dynamics and through the rendering process --
enables learning in challenging visuomotor control tasks, without relying on
state-based (3D) supervision, while obtaining performance competitive to or
better than techniques that rely on precise 3D labels.
