Original: ∇ Sim : D IFFERENTIABLE SIMULATION FOR SYSTEM IDENTIFICATION AND VISUOMOTOR CONTROL
Revised: ∇ Sim : D IFFERENTIABLE SIMULATION FOR SYSTEM IDENTIFICATION AND VISUOMOTOR CONTROL

Original: Anonymous authors
Revised: https://gradsim.github.io

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We consider the problem of estimating object physical properties such as mass, friction, and elasticity directly from video sequences.
Revised: We consider the problem of estimating an object’s physical properties such as mass, friction, and elasticity directly from video sequences.

Original: Such a system identiﬁcation problem is fundamentally ill-posed due to the loss of information during image formation.
Revised: Such a system identiﬁcation problem is fundamentally ill-posed due to the loss of information during image formation.

Original: Current solutions require precise 3D labels which are labor-intensive to gather, and infeasible to create for many systems such as deformable solids or cloth.
Revised: Current solutions require precise 3D labels which are labor-intensive to gather, and infeasible to create for many systems such as deformable solids or cloth.

Original: We present ∇ Sim , a framework that overcomes the dependence on 3D supervision by leveraging differentiable multiphysics simulation and differentiable rendering to jointly model the evolution of scene dynamics and image formation.
Revised: We present ∇ Sim , a framework that overcomes the dependence on 3D supervision by leveraging differentiable multiphysics simulation and differentiable rendering to jointly model the evolution of scene dynamics and image formation.

Original: This novel combination enables backpropagation from pixels in a video sequence through to the underlying physical attributes that generated them.
Revised: This novel combination enables backpropagation from pixels in a video sequence through to the underlying physical attributes that generated them.

Original: Moreover, our uniﬁed computation graph – spanning from the dynamics and through the rendering process – enables learning in challenging visuomotor control tasks, without relying on state-based (3D) supervision, while obtaining performance competitive to or better than techniques that rely on precise 3D labels.
Revised: Moreover, our uniﬁed computation graph – spanning from the dynamics and through the rendering process – enables learning in challenging visuomotor control tasks, without relying on state-based (3D) supervision, while obtaining performance competitive to or better than techniques that rely on precise 3D labels.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Accurately predicting the dynamics and physical characteristics of objects from image sequences is a long-standing challenge in computer vision.
Revised: Accurately predicting the dynamics and physical characteristics of objects from image sequences is a long-standing challenge in computer vision.

Original: Doing so enables higher-level end-to-end reasoning, but requires a fundamental understanding of both the underlying physical system and image formation process.
Revised: This end-to-end reasoning task requires a fundamental understanding of both the underlying scene dynamics and the imaging process.

Original: Imagine watching a short video of a basketball bouncing off the ground and ask: is it possible to infer the mass and elasticity of the ball in order to predict its trajectory and make informed decisions, e.g., how to pass and shoot?
Revised: Imagine watching a short video of a basketball bouncing off the ground and ask: “Can we infer the mass and elasticity of the ball, predict its trajectory, and make informed decisions, e.g., how to pass and shoot?”

Original: This seemingly simple question is extremely challenging to answer with modern computer vision models, as the physicaland material attributes of objects and theunderlying system dynamics need to be modeled and estimated,and all while accounting for the loss of information during 3D to 2D image formation.
Revised: These seemingly simple questions are extremely challenging to answer even for modern computer vision models. The underlying physical attributes of objects and the system dynamics need to be modeled and estimated, all while accounting for the loss of information during 3D to 2D image formation.

Original: Recent solutions fall into three categories – black , grey , or white box – depending on their assumptions on the system’s structure.
Revised: Depending on the assumptions on the scene structre and dynamics, three types of solutions exist: black , grey , or white box .

Original: Black box methods (Watters et al., 2017; Xu et al., 2019b; Janner et al., 2019;Ehrhardt et al., 2017; Chang et al., 2016) model the state of a dynamical system (such as the basketball’s trajectory in time) as a learned embedding of its visual representation and dynamics.
Revised: Black box methods (Watters et al., 2017; Xu et al., 2019b; Janner et al., 2019; Chang et al., 2016) model the state of a dynamical system (such as the basketball’s trajectory in time) as a learned embedding of its states or observations.

Original: These methods require few prior assumptions about the system itself, but lack interpretability due inherent entangling of variational factors (Chen et al., 2016) or due to the ambiguitiespresent in unsupervised learning (Greydanus et al., 2019; Cranmer et al., 2020b).
Revised: These methods require few prior assumptions about the system itself, but lack interpretability due to entangled variational factors (Chen et al., 2016) or due to the ambiguities in unsupervised learning (Greydanus et al., 2019; Cranmer et al., 2020b).

Original: More recently, grey box methods (Mehta et al., 2020) propose to leverage prior knowledge of the system dynamics to improve learning.
Revised: Recently, grey box methods (Mehta et al., 2020) leveraged partial knowledge about the system dynamics to improve performance.

Original: White box methods (Degrave et al., 2016;de Avila Belbute-Peres et al., 2018; Liang et al., 2019;
Revised: In contrast, white box methods (Degrave et al., 2016;

Original:  Hu et al., 2020; Qiao et al., 2020)similarly impose prior knowledge, albet in a more accurate and prescriptive manner: explicit dynamics models are employed, reducing the space of learnable parameters to a few unspeciﬁed parameters.
Revised: Liang et al., 2019; Hu et al., 2020; Qiao et al., 2020) impose prior knowledge by employing explicit dynamics models, reducing the space of learnable parameters and improving system interpretability.

Original: Most notably in our context, all of these approaches require precise 3D labels – which are labor intensive to gather, and infeasible to generate for many systems such as deformable solids or cloth.
Revised: Most notably in our context, all of these approaches require precise 3D labels – which are laborintensive to gather, and infeasible to generate for many systems such as deformable solids or cloth.

Original: We eliminate the dependence of white box dynamics methods on 3D supervision by coupling explicit (and differentiable) models of scene dynamics withdifferentiable image formation (rendering) 1 .
Revised: We eliminate the dependence of white box dynamics methods on 3D supervision by coupling explicit (and differentiable) models of scene dynamics with image formation (rendering) 1 .

Original: Explicitly modeling the end-to-end dynamics and image formation underlying video observations is challenging, even with access to the full system state.
Revised: Explicitly modeling the end-to-end dynamics and image formation underlying video observations is challenging, even with access to the full system state.

Original: This problem has been treated in the vision, graphics and physics communities (Pharr et al., 2016; Macklin et al., 2014), leading to the development of robust forward simulation models and algorithms.
Revised: This problem has been treated in the vision, graphics, and physics communities (Pharr et al., 2016; Macklin et al., 2014), leading to the development of robust forward simulation models and algorithms.

Original: These simulators are not readily amenable to solving inverse problems, due in part to their non-differentiability.
Revised: These simulators are not readily usable for solving inverse problems, due in part to their non-differentiability.

Original: As such, applications ofblack-box forward processes often require surrogate gradient estimators such as ﬁnite differences or REINFORCE (Williams, 1992) to enable any learning.
Revised: As such, applications of black-box forward processes often require surrogate gradient estimators such as ﬁnite differences or REINFORCE (Williams, 1992) to enable any learning.

Original: Likelihood-free inference for black-box forward simulators (Ramos et al., 2019; Cranmer et al., 2020a; Kulkarni et al., 2015; Yildirim et al., 2017; 2015; 2020; Wu et al., 2017b) has led to some improvements here, but remains limited in terms of data efﬁciency and scalability to high dimensional parameter spaces.
Revised: Likelihood-free inference for black-box forward simulators (Ramos et al., 2019; Cranmer et al., 2020a; Kulkarni et al., 2015; Yildirim et al., 2017; 2015; 2020; Wu et al., 2017b) has led to some improvements here, but remains limited in terms of data efﬁciency and scalability to high dimensional parameter spaces.

Original: Recent progress in differentiable simulation further improves the learning dynamics, however we still lack a method for end-to-end differentiation through the entire simulation process (i.e., from video pixels to physical attributes), a prerequisite for effective learning from video frames alone.
Revised: Recent progress in differentiable simulation further improves the learning dynamics, however we still lack a method for end-to-end differentiation through the entire simulation process (i.e., from video pixels to physical attributes), a prerequisite for effective learning from video frames alone.

Original: We present ∇ Sim , a versatile end-to-end differentiable simulator that adopts a holistic, uniﬁed view of differentiable dynamics and image formation ( cf.
Revised: We present ∇ Sim , a versatile end-to-end differentiable simulator that adopts a holistic, uniﬁed view of differentiable dynamics and image formation( cf.

Original: Fig.1).
Revised: Fig.

Original: Existing differentiable physics engines only model time-varying dynamics and require supervision in state space (usually 3D tracking).
Revised: Existing differentiable physics engines only model time-varying dynamics and require supervision in state space (usually 3D tracking).

Original: We additionally model a differentiable image formation process, thus only requiring target information speciﬁed in image space .
Revised: We additionally model a differentiable image formation process, thus only requiring target information speciﬁed in image space .

Original: This enables us to backpropagate (Griewank & Walther, 2003) training signals from video pixels all the way to the underlying physical and dynamical attributes of a scene.
Revised: This enables us to backpropagate (Griewank & Walther, 2003) training signals from video pixels all the way to the underlying physical and dynamical attributes of a scene.

Original: Our main contributions are:
Revised: Our main contributions are:

Original: • ∇ Sim , a differentiable simulator that demonstrates the ability to backprop from video pixels to the underlying physical attributes ( cf.
Revised: • ∇ Sim , a differentiable simulator that demonstrates the ability to backprop from video pixels to the underlying physical attributes ( cf.

Original: Fig.2).
Revised: Fig.

Original: • We demonstrate recovering many physical properties exclusively from video observations, including friction, elasticity, deformable material parameters, and visuomotor controls (sans 3D supervision)
Revised: • We demonstrate recovering many physical properties exclusively from video observations, including friction, elasticity, deformable material parameters, and visuomotor controls (sans 3D supervision)

Original: • A(soon-to-be) publicly available PyTorch framework, facilitating interoperability with existing machine learning modules and imperative programming.
Revised: • A PyTorch framework facilitating interoperability with existing machine learning modules.

Original: We evaluate ∇ Sim ’s effectiveness on parameter identiﬁcation tasks for rigid, deformable and thin-shell bodies, and demonstrate performance  competitive, or in some cases superior, to current physics-only differentiable simulators.
Revised: We evaluate ∇ Sim ’s effectiveness on parameter identiﬁcation tasks for rigid, deformable and thinshell bodies, and demonstrate performance that is competitive, or in some cases superior, to current physics-only differentiable simulators.

Original: Additionally, we demonstrate the effectiveness of the gradients provided by ∇ Sim on challenging visuomotor control tasks involving deformable solids and cloth.
Revised: Additionally, we demonstrate the effectiveness of the gradients provided by ∇ Sim on challenging visuomotor control tasks involving deformable solids and cloth.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Differentiable physics simulators have seen signiﬁcant attention and activity, with efforts centered around embedding physics structure into autodifferentiation frameworks.
Revised: Differentiable physics simulators have seen signiﬁcant attention and activity, with efforts centered around embedding physics structure into autodifferentiation frameworks.

Original: This has enabled differentiation through contact and friction models (Toussaint et al., 2018;de Avila Belbute-Peres et al., 2018; Song & Boularias, 2020b;a; Degrave et al., 2016; Wu et al., 2017a; Research, 2020 (accessed
Revised: This has enabled differentiation through contact and friction models (Toussaint et al., 2018; Belbute-Peres et al., ; Song & Boularias, 2020b;a; Degrave et al., 2016; Wu et al., 2017a; Research, 2020 (accessed

Original: May 15, 2020), latent state models (Guen & Thome, 2020; Schenck & Fox, 2018;
Revised: May 15, 2020), latent state models (Guen & Thome, 2020; Schenck & Fox, 2018;

Original: Jaques et al., 2020;
Revised: Jaques et al., 2020;

Original: Heiden et al., 2019), volumetric soft bodies (Hu et al., 2019; 2018;
Revised: Heiden et al., 2019), volumetric soft bodies (Hu et al., 2019; 2018;

Original: Liang et al., 2019; Hu et al., 2020), as well as particle dynamics (Schenck & Fox, [-
 ;-] Li et al., 2019; 2020; Hu et al., 2020).
Revised: Liang et al., 2019; Hu et al., 2020), as well as particle dynamics (Schenck & Fox, [-
 ;-]2018; Li et al., 2019; 2020; Hu et al., 2020).

Original: In contrast, ∇ Sim addresses a superset of simulation scenarios, by coupling the physics simulator with a differentiable rendering pipeline.
Revised: In contrast, ∇ Sim addresses a superset of simulation scenarios, by coupling the physics simulator with a differentiable rendering pipeline.

Original: It also supports tetrahedral FEM-based hyperelasticity models to simulate deformable solids and thin-shells.
Revised: It also supports tetrahedral FEM-based hyperelasticity models to simulate deformable solids and thin-shells.

Original: Recent work on physics-based deep learning injects structure in the latent space of the dynamics using Lagrangian and Hamiltonian operators (Greydanus et al., 2019; Chen et al., 2020; Toth et al., 2020; Sanchez-Gonzalez et al., 2019; Cranmer et al., 2020b; Zhong et al., 2020), by explicitly conserving physical quantities, or with ground truth supervision (Asenov et al., 2019; Wu et al., 2016;
Revised: Recent work on physics-based deep learning injects structure in the latent space of the dynamics using Lagrangian and Hamiltonian operators (Greydanus et al., 2019; Chen et al., 2020; Toth et al., 2020; Sanchez-Gonzalez et al., 2019; Cranmer et al., 2020b; Zhong et al., 2020), by explicitly conserving physical quantities, or with ground truth supervision (Asenov et al., 2019; Wu et al., 2016;

Original: Xu et al., 2019b).
Revised: Xu et al., 2019b).

Original: Sensor readings have been used to predicting the effects of forces applied to an object in models of learned (Fragkiadaki et al., 2016; Byravan & Fox, 2017) and intuitive physics (Ehsani et al., 2020;
Revised: Sensor readings have been used to predicting the effects of forces applied to an object in models of learned (Fragkiadaki et al., 2016; Byravan & Fox, 2017) and intuitive physics (Ehsani et al., 2020;

Original: Mottaghi et al., 2015; 2016; Gupta et al., 2010; Ehrhardt et al., 2018; Yu et al., 2015; Battaglia et al., 2013; Mann et al., 1997; Innamorati et al., 2019; Standley et al., 2017).
Revised: Mottaghi et al., 2015; 2016; Gupta et al., 2010; Ehrhardt et al., 2018; Yu et al., 2015; Battaglia et al., 2013; Mann et al., 1997; Innamorati et al., 2019; Standley et al., 2017).

Original: This also includes approaches that learn to model multi-object interactions (Watters et al., 2017; Xu et al., 2019b; Janner et al., 2019;
Revised: This also includes approaches that learn to model multi-object interactions (Watters et al., 2017; Xu et al., 2019b; Janner et al., 2019;

Original: Ehrhardt et al., 2017; Chang et al., 2016; Agrawal et al., 2016).
Revised: Ehrhardt et al., 2017; Chang et al., 2016; Agrawal et al., 2016).

Original: In many cases, intuitive physics approaches are limited in their prediction horizon and treatment of complex scenes, as they do not sufﬁciently accurately model the 3D geometry nor the object properties.
Revised: In many cases, intuitive physics approaches are limited in their prediction horizon and treatment of complex scenes, as they do not sufﬁciently accurately model the 3D geometry nor the object properties.

Original: System identiﬁcation based on parameterized physics models (Salzmann & Urtasun, 2011; Brubaker et al., 2010; Kozlowski, 1998; Wensing et al., 2018; Brubaker et al., 2009;
Revised: System identiﬁcation based on parameterized physics models (Salzmann & Urtasun, 2011; Brubaker et al., 2010; Kozlowski, 1998; Wensing et al., 2018; Brubaker et al., 2009;

Original: Bhat et al., 2003; 2002; Liu et al., 2005; Grzeszczuk et al., 1998; Sutanto et al., 2020; Wang et al., 2020; 2018a) and inverse simulation (Murray-Smith, 2000) are closely related areas.
Revised: Bhat et al., 2003; 2002; Liu et al., 2005; Grzeszczuk et al., 1998; Sutanto et al., 2020; Wang et al., 2020; 2018a) and inverse simulation (Murray-Smith, 2000) are closely related areas.

Original: There is a rich literature on neural image synthesis , but we focus on methods that model the 3D scene structure, including voxels (Henzler et al., 2019; Paschalidou et al., 2019; Smith et al., 2018b;
Revised: There is a rich literature on neural image synthesis , but we focus on methods that model the 3D scene structure, including voxels (Henzler et al., 2019; Paschalidou et al., 2019; Smith et al., 2018b;

Original: Nguyen-Phuoc et al., 2018), meshes (Smith et al., 2020;
Revised: Nguyen-Phuoc et al., 2018), meshes (Smith et al., 2020;

Original: Wang et al., 2018b; Groueix et al., 2018;
Revised: Wang et al., 2018b; Groueix et al., 2018;

Original: Alhaija et al., 2018), and implicit shapes (Xu et al., 2019a; Chen & Zhang, 2019; Michalkiewicz et al., 2019; Niemeyer et al., 2020; Park et al., 2019; Mescheder et al., 2019).
Revised: Alhaija et al., 2018), and implicit shapes (Xu et al., 2019a; Chen & Zhang, 2019; Michalkiewicz et al., 2019; Niemeyer et al., 2020; Park et al., 2019; Mescheder et al., 2019).

Original: Generative models condition the rendering process on samples of the 3D geometry (Liao et al., 2019).
Revised: Generative models condition the rendering process on samples of the 3D geometry (Liao et al., 2019).

Original: Latent factors determining 3D structure have also been learned in generative models (Chen et al., 2016; Eslami et al., 2018).
Revised: Latent factors determining 3D structure have also been learned in generative models (Chen et al., 2016; Eslami et al., 2018).

Original: Additionally, implicit neural representations that leverage differentiable rendering have been proposed (Mildenhall et al., 2020; 2019) for realistic view synthesis.
Revised: Additionally, implicit neural representations that leverage differentiable rendering have been proposed (Mildenhall et al., 2020; 2019) for realistic view synthesis.

Original: Many of these representations have become easy to manipulate through software frameworks like Kaolin (Jatavallabhula et al., 2019),
Revised: Many of these representations have become easy to manipulate through software frameworks like Kaolin (Jatavallabhula et al., 2019),

Original: Open3D (Zhou et al., 2018), and PyTorch3D (Ravi et al., 2020).
Revised: Open3D (Zhou et al., 2018), and PyTorch3D (Ravi et al., 2020).

Original: Differentiable rendering allows for image gradients to be computed w.r.t.
Revised: Differentiable rendering allows for image gradients to be computed w.r.t.

Original: the scene geometry, camera, and lighting inputs.
Revised: the scene geometry, camera, and lighting inputs.

Original: Variants based on the rasterization paradigm (NMR (Kato et al., 2018),
Revised: Variants based on the rasterization paradigm (NMR (Kato et al., 2018),

Original: OpenDR (Loper & Black, 2014), SoftRas (Liu et al., 2019)) blur the edges of scene triangles prior to image projection to remove discontinuities in the rendering signal.
Revised: OpenDR (Loper & Black, 2014), SoftRas (Liu et al., 2019)) blur the edges of scene triangles prior to image projection to remove discontinuities in the rendering signal.

Original: DIB-R
Revised: DIB-R

Original: (Chen et al., 2019) applies this idea to background pixels and proposes an interpolation-based rasterizer for foreground pixels.
Revised: (Chen et al., 2019) applies this idea to background pixels and proposes an interpolation-based rasterizer for foreground pixels.

Original: More sophisticated differentiable renderers can treat physics-based light transport processes (Li et al., 2018; Nimier-David et al., 2019) by ray tracing, and more readily support higher-order effects such as shadows, secondary light bounces, and global illumination.
Revised: More sophisticated differentiable renderers can treat physics-based light transport processes (Li et al., 2018; Nimier-David et al., 2019) by ray tracing, and more readily support higher-order effects such as shadows, secondary light bounces, and global illumination.

Original:  ∇ Sim : A UNIFIED DIFFERENTIABLE SIMULATION ENGINE
Revised:  ∇ Sim : A UNIFIED DIFFERENTIABLE SIMULATION ENGINE

Original: Typically, physics estimation and rendering have been treated as disjoint, mutually exclusive tasks.
Revised: Typically, physics estimation and rendering have been treated as disjoint, mutually exclusive tasks.

Original: In this work, we take on a uniﬁed view of simulation in general, to compose physics estimation and rendering.
Revised: In this work, we take on a uniﬁed view of simulation in general, to compose physics estimation and rendering.

Original: Formally, simulation is a function Sim : R P × [0 , 1] (cid:55)→ R H × R W ; Sim ( p , t )
Revised: Formally, simulation is a function Sim : R P × [0 , 1] (cid:55)→ R H × R W ; Sim ( p , t )

Original: =
Revised: =

Original: I .
Revised: I .

Original: Here p ∈ R P is a vector representing the simulation state and parameters (objects, their physical properties, their geometries, etc.), t denotes the time of simulation (conveniently reparameterized to be in the interval [0 , 1] ).
Revised: Here p ∈ R P is a vector representing the simulation state and parameters (objects, their physical properties, their geometries, etc.), t denotes the time of simulation (conveniently reparameterized to be in the interval [0 , 1] ).

Original: Given initial conditions p 0 , the simulation function produces an image
Revised: Given initial conditions p 0 , the simulation function produces an image

Original: I of height H and width W at each timestep t .
Revised: I of height H and width W at each timestep t .

Original: If this function Sim were differentiable, then the gradient of Sim ( p , t ) with respect to the simulation parameters p provides the change in the output of the simulation from I to I + ∇ Sim ( p , t ) δ p due to an inﬁnitesimal perturbation of p by δ p .
Revised: If this function Sim were differentiable, then the gradient of Sim ( p , t ) with respect to the simulation parameters p provides the change in the output of the simulation from I to I + ∇ Sim ( p , t ) δ p due to an inﬁnitesimal perturbation of p by δ p .

Original: This construct enables a gradient-based optimizer to estimate physical parameters from video , by deﬁning a loss function over the image space L ( I , . ) , and descending this loss landscape along a direction parallel to −∇ Sim ( . ) .
Revised: This construct enables a gradient-based optimizer to estimate physical parameters from video , by deﬁning a loss function over the image space L ( I , . ) , and descending this loss landscape along a direction parallel to −∇ Sim ( . ) .

Original: To realise this, we turn to the paradigms of computational graphs and differentiable programming .
Revised: To realise this, we turn to the paradigms of computational graphs and differentiable programming .

Original: ∇ Sim comprises two main components: a differentiable physics engine that computes the physical states of the scene at each time instant, and a differentiable renderer that renders the scene to a 2D image.
Revised: ∇ Sim comprises two main components: a differentiable physics engine that computes the physical states of the scene at each time instant, and a differentiable renderer that renders the scene to a 2D image.

Original: Contrary to existing differentiable physics (Toussaint et al., 2018; de Avila Belbute-Peres et al., 2018; Song & Boularias, 2020b;a; Degrave et al., 2016; Wu et al., 2017a;
Revised: Contrary to existing differentiable physics (Toussaint et al., 2018; de Avila Belbute-Peres et al., 2018; Song & Boularias, 2020b;a; Degrave et al., 2016; Wu et al., 2017a;

Original: Research,(accessed May 15, 2020; Hu et al., 2020; Qiao et al., 2020) or differentiable rendering (Loper &
Revised: Research,(accessed May 15, 2020; Hu et al., 2020; Qiao et al., 2020) or differentiable rendering (Loper &

Original: Black, 2014; Kato et al., 2018; Liu et al., 2019; Chen et al., 2019) approaches, we adopt a holistic view and construct a computational graph spanning them both.
Revised: Black, 2014; Kato et al., 2018; Liu et al., 2019; Chen et al., 2019) approaches, we adopt a holistic view and construct a computational graph spanning them both.

Original: 3. D IFFERENTIABLE PHYSICS ENGINE
Revised: 3. D IFFERENTIABLE PHYSICS ENGINE

Original: Under Lagrangian mechanics, the state of a physical system can be described in terms of generalized coordinates q , generalized velocities ˙ q = u , and design/model parameters θ .
Revised: Under Lagrangian mechanics, the state of a physical system can be described in terms of generalized coordinates q , generalized velocities ˙ q = u , and design/model parameters θ .

Original: For the purpose of exposition, we make no distinction between rigid bodies, or deformable solids, or thin-shell models of cloth, etc.
Revised: For the purpose of exposition, we make no distinction between rigid bodies, or deformable solids, or thin-shell models of cloth, etc.

Original: Although the speciﬁc choices of coordinates and parameters vary, the simulation procedure is virtually unchanged.
Revised: Although the speciﬁc choices of coordinates and parameters vary, the simulation procedure is virtually unchanged.

Original: We denote the combined state vector by s ( t ) =
Revised: We denote the combined state vector by s ( t ) =

Original: [ q ( t ) , u ( t )] .
Revised: [ q ( t ) , u ( t )] .

Original: The dynamic evolution of the system is governed by second order differential equations (ODEs) of the form M ( s , θ )¨ s =f
Revised: The dynamic evolution of the system is governed by second order differential equations (ODEs) of the form M ( s , θ )˙ s =

Original:  ( s , θ ) , where M is a mass matrix that depends on the state and parameters.
Revised: f ( s , θ ) , where M is a mass matrix that depends on the state and parameters.

Original: The forces on the system may be parameterized by design parameters (e.g. Young’s modulus).
Revised: The forces on the system may be parameterized by design parameters (e.g. Young’s modulus).

Original: Solutions to these ODEs may be obtained through black box numerical integration methods, and their derivatives calculated through the continuous adjoint method (Chen et al., 2018).
Revised: Solutions to these ODEs may be obtained through black box numerical integration methods, and their derivatives calculated through the continuous adjoint method (Chen et al., 2018).

Original: However, we instead consider our physics engine as a differentiable operation that provides an implicit relationship between a state vector s − =
Revised: However, we instead consider our physics engine as a differentiable operation that provides an implicit relationship between a state vector s − =

Original: s ( t ) at the start of a time step, and the updated state at the end of the time step s + = s ( t + ∆ t ) .
Revised: s ( t ) at the start of a time step, and the updated state at the end of the time step s + = s ( t + ∆ t ) .

Original: An arbitrary discrete time integration scheme can be then be abstracted as the function g ( s − , s + , θ ) = 0 , relating the initial and ﬁnal system state and the model parameters θ .
Revised: An arbitrary discrete time integration scheme can be then be abstracted as the function g ( s − , s + , θ ) = 0 , relating the initial and ﬁnal system state and the model parameters θ .

Original: Gradients through this dynamical system can be computed by graph-based autodiff frameworks (Paszke et al., 2019; Abadi et al., 2015;
Revised: Gradients through this dynamical system can be computed by graph-based autodiff frameworks (Paszke et al., 2019; Abadi et al., 2015;

Original: Bradbury et al., 2018), or by program transformation approaches (Hu et al., 2020; van Merriënboer et al., 2018).
Revised: Bradbury et al., 2018), or by program transformation approaches (Hu et al., 2020; van Merriënboer et al., 2018).

Original: Our framework is agnostic to the speciﬁcs of the differentiable physics engine, however in Appendices A through D we detail an efﬁcient approach based on the source-code transformation of parallel kernels, similar to DiffTaichi (Hu et al., 2020).
Revised: Our framework is agnostic to the speciﬁcs of the differentiable physics engine, however in Appendices A through D we detail an efﬁcient approach based on the source-code transformation of parallel kernels, similar to DiffTaichi (Hu et al., 2020).

Original: In addition, we describe extensions to this framework to support mesh-based tetrahedral ﬁnite-element models (FEMs) for deformable and thin-shell solids.
Revised: In addition, we describe extensions to this framework to support mesh-based tetrahedral ﬁnite-element models (FEMs) for deformable and thin-shell solids.

Original: This is important since we require surface meshes to perform differentiable rasterization as described in the following section.
Revised: This is important since we require surface meshes to perform differentiable rasterization as described in the following section.

Original: 3. D IFFERENTIABLE RENDERING ENGINE
Revised: 3. D IFFERENTIABLE RENDERING ENGINE

Original: A renderer expects scene description inputs and generates color image outputs, all according to a sequence of image formation stages deﬁned by the forward graphics pipeline.
Revised: A renderer expects scene description inputs and generates color image outputs, all according to a sequence of image formation stages deﬁned by the forward graphics pipeline.

Original: The scene description includes a complete geometric descriptor of scene elements, their associated material/reﬂectance properties, light source deﬁnitions, and virtual camera parameters.
Revised: The scene description includes a complete geometric descriptor of scene elements, their associated material/reﬂectance properties, light source deﬁnitions, and virtual camera parameters.

Original: The rendering process is not generally differentiable, as visibility and occlusion events introduce discontinuities.
Revised: The rendering process is not generally differentiable, as visibility and occlusion events introduce discontinuities.

Original: Most interactive renderers, such as those used in real-time applications, employ a rasterization process to project 3D geometric primitives onto 2D pixel coordinates, resolving these visibility events with nondifferentiable operations.
Revised: Most interactive renderers, such as those used in real-time applications, employ a rasterization process to project 3D geometric primitives onto 2D pixel coordinates, resolving these visibility events with nondifferentiable operations.

Original: Our experiments employ two differentiable alternatives to traditional rasterization, SoftRas (Liu et al., 2019) and DIB-R (Chen et al., 2019), both of which replace discontinuous triangle mesh edges with smooth sigmoids.
Revised: Our experiments employ two differentiable alternatives to traditional rasterization, SoftRas (Liu et al., 2019) and DIB-R (Chen et al., 2019), both of which replace discontinuous triangle mesh edges with smooth sigmoids.

Original: This has the effect of blurring triangle edges into semi-transparent boundaries, thereby removing the non-differentiable discontinuity of traditional rasterization.
Revised: This has the effect of blurring triangle edges into semi-transparent boundaries, thereby removing the non-differentiable discontinuity of traditional rasterization.

Original: DIBR distinguishes between foreground pixels (associated to the principal object being rendered in the scene) and background pixels (for all other objects, if any).
Revised: DIBR distinguishes between foreground pixels (associated to the principal object being rendered in the scene) and background pixels (for all other objects, if any).

Original: The latter are rendered using the same technique as SoftRas while the former are rendered by bilinearly sampling a texture using differentiable UV coordinates.
Revised: The latter are rendered using the same technique as SoftRas while the former are rendered by bilinearly sampling a texture using differentiable UV coordinates.

Original: ∇ Sim performs differentiable physics simulation and rendering at independent and adjustable rates, allowing us to trade computation for accuracy by rendering fewer frames than dynamics updates.
Revised: ∇ Sim performs differentiable physics simulation and rendering at independent and adjustable rates, allowing us to trade computation for accuracy by rendering fewer frames than dynamics updates.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We conducted multiple experiments to test the efﬁcacy of ∇ Sim on physical parameter identiﬁcation from video and visuomotor control , to address the following questions:
Revised: We conducted multiple experiments to test the efﬁcacy of ∇ Sim on physical parameter identiﬁcation from video and visuomotor control , to address the following questions:

Original: • Can we accurately identify physical parameters by backpropagating from video pixels, through the simulator?
Revised: • Can we accurately identify physical parameters by backpropagating from video pixels, through the simulator?

Original: (Ans: Yes, very accurately ; cf. 4.1) • What is the performance gap associated with using ∇ Sim (2D supervision) vs. differentiable physics-only engines (3D supervision)?
Revised: (Ans: Yes, very accurately , cf. 4.1) • What is the performance gap associated with using ∇ Sim (2D supervision) vs. differentiable physics-only engines (3D supervision)?

Original: (Ans: ∇ Sim is competitive/superior ; cf.
Revised: (Ans: ∇ Sim is competitive/superior , cf.

Original: Tables 1, 2, 3) • How do loss landscapes differ across differentiable simulators ( ∇ Sim ) and their non-differentiable counterparts?
Revised: Tables 1, 2, 3) • How do loss landscapes differ across differentiable simulators ( ∇ Sim ) and their non-differentiable counterparts?

Original: (Ans: Loss landscapes for ∇ Sim are smooth ; cf. 4.1.3) • Can we use ∇ Sim for visuomotor control tasks?
Revised: (Ans: Loss landscapes for ∇ Sim are smooth , cf. 4.1.3) • Can we use ∇ Sim for visuomotor control tasks?

Original: (Ans: Yes, without any 3D supervision ; cf. 4.2) • How sensitive is ∇ Sim to modeling assumptions at system level?
Revised: (Ans: Yes, without any 3D supervision , cf. 4.2) • How sensitive is ∇ Sim to modeling assumptions at system level?

Original: (Ans: Moderate ; cf.
Revised: (Ans: Moderately , cf.

Original: Table 4)
Revised: Table 4)

Original: Each of our experiments comprises an environment E that applies a particular set of physical forces and/or constraints, a (differentiable) loss function L that implicitly speciﬁes an objective, and an initial guess θ 0 of the physical state of the simulation.
Revised: Each of our experiments comprises an environment E that applies a particular set of physical forces and/or constraints, a (differentiable) loss function L that implicitly speciﬁes an objective, and an initial guess θ 0 of the physical state of the simulation.

Original: The goal is to recover optimal physics parameters θ ∗ that minimize L , by backpropagating through the simulator.
Revised: The goal is to recover optimal physics parameters θ ∗ that minimize L , by backpropagating through the simulator.

Original: 4. P HYSICAL PARAMETER ESTIMATION FROM VIDEO
Revised: 4. P HYSICAL PARAMETER ESTIMATION FROM VIDEO

Original: First, we assess the capabilities of ∇ Sim to accurately identify a variety of physical attributes such as mass, friction, and elasticity from image/video observations.
Revised: First, we assess the capabilities of ∇ Sim to accurately identify a variety of physical attributes such as mass, friction, and elasticity from image/video observations.

Original: To the best of our knowledge, ∇ Sim is the ﬁrst study to jointly infer such ﬁne-grained parameters from video observations.
Revised: To the best of our knowledge, ∇ Sim is the ﬁrst study to jointly infer such ﬁne-grained parameters from video observations.

Original: We also implement a set of competitive baselines that use strictly more information on the task.
Revised: We also implement a set of competitive baselines that use strictly more information on the task.

Original: 4.1. R IGID BODIES ( RIGID )
Revised: 4.1. R IGID BODIES ( RIGID )

Revised: we optimize the material properties of a beam to match a video of a beam hanging under gravity.

Revised: In the rigid experiments, we estimate contact parameters (elasticity/friction) and object density to match a video (GT).

Revised: We visualize entire time sequences

Original: Our ﬁrst environment– rigid –evaluates the accuracy of estimating of physical and material attributes of rigid objects from videos.
Revised: Our ﬁrst environment– rigid –evaluates the accuracy of estimating of physical and material attributes of rigid objects from videos.

Original: We curate a dataset of 10000 simulated videos generated from variations of 14 objects, comprising primitive shapes such as boxes, cones, cylinders, as well as nonconvex shapes from ShapeNet (Chang et al., 2015) and DexNet (Mahler et al., 2017).
Revised: We curate a dataset of 10000 simulated videos generated from variations of 14 objects, comprising primitive shapes such as boxes, cones, cylinders, as well as non-convex shapes from ShapeNet (Chang et al., 2015) and DexNet (Mahler et al., 2017).

Original: With uniformly sampled initial dimensions, poses, velocities, and physical properties (density, elasticity, and friction parameters), we apply a known impulse to the object and record a video of the resultant trajectory.
Revised: With uniformly sampled initial dimensions, poses, velocities, and physical properties (density, elasticity, and friction parameters), we apply a known impulse to the object and record a video of the resultant trajectory.

Original: Inference with ∇ Sim is done by picking an initialguess of the mass (uniformly randomly in the range [2 , 12] kg/m 3 ), unrolling a differentiable simulation using this guess, comparing the rendered out video with the true video (pixelwise mean-squared errorMSE), and performing gradient descent updates.
Revised: Inference with ∇ Sim is done by guessing an initial mass (uniformly random in the range [2 , 12] kg/m 3 ), unrolling a differentiable simulation using this guess, comparing the rendered out video with the true video (pixelwise mean-squared error - MSE), and performing gradient descent updates.

Original: We refer the interested reader to the appendix (Sec. G) for more details.
Revised: We refer the interested reader to the appendix (Sec. G) for more details.

Original: Table 1 shows the results for predicting the mass of an object from video, with a known impulse applied to it.
Revised: Table 1 shows the results for predicting the mass of an object from video, with a known impulse applied to it.

Original: We use EfﬁcientNet (B0)
Revised: We use EfﬁcientNet (B0)

Original: (Tan & Le, 2019) and resize input frames to 64 × 64 .
Revised: (Tan & Le, 2019) and resize input frames to 64 × 64 .

Original: Feature maps at a resoluition of 4 × 4 × 32 are concatenated for all frames and fed to an MLP with 4 linear layers, and trained with an MSE loss.
Revised: Feature maps at a resoluition of 4 × 4 × 32 are concatenated for all frames and fed to an MLP with 4 linear layers, and trained with an MSE loss.

Original: We compare ∇ Sim with three other baselines:
Revised: We compare ∇ Sim with three other baselines:

Original: PyBullet + REINFORCE (Ehsani et al., 2020; Wu et al., 2015), diff.
Revised: PyBullet + REINFORCE (Ehsani et al., 2020; Wu et al., 2015), diff.

Original: physics only (requiring 3D supervision), and a ConvLSTM baseline adopted from Xu et al.
Revised: physics only (requiring 3D supervision), and a ConvLSTM baseline adopted from Xu et al.

Original: (2019b)
Revised: (2019b)

Original: but with a stronger backbone.
Revised: but with a stronger backbone.

Original: The DiffPhysics baseline is a strict subset of ∇ Sim , it only inolves the differentiable physics engine.
Revised: The DiffPhysics baseline is a strict subset of ∇ Sim , it only inolves the differentiable physics engine.

Original: However, it needs precise 3D states as supervision, which is the primary factor for its superior performance.
Revised: However, it needs precise 3D states as supervision, which is the primary factor for its superior performance.

Original: Nevertheless, ∇ Sim is able to very precisely estimate mass from video, to a absolute relative error of 9.01e-5, nearly 2 orders of magnitude better than the ConvLSTM baseline.
Revised: Nevertheless, ∇ Sim is able to very precisely estimate mass from video, to a absolute relative error of 9.01e-5, nearly two orders of magnitude better than the ConvLSTM baseline.

Original: We also use two other baselines: the “ Average ” baseline always predicts the dataset mean; the “ Random ” baselinepredicts a random parameter value from the test distribution. We explain all baselines and
Revised: Two other baselines are also used: the “ Average ” baseline always predicts the dataset mean and the “ Random ” baseline

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Revised: predicts a random parameter value from the test distribution.

Revised: All baselines and training details can be found in Sec.

Original:  To investigate whether analytical differentiability is required, our PyBullet + REINFORCE baseline applies black-box gradient estimation (Williams, 1992) through a non-differentiable simulator (Coumans & Bai, 2016–2019), similar to Ehsani et al.
Revised: H of the appendix. To investigate whether analytical differentiability is required, our PyBullet + REINFORCE baseline applies black-box gradient estimation (Williams, 1992) through a non-differentiable simulator (Coumans & Bai, 2016–2019), similar to Ehsani et al.

Original: We ﬁnd this baseline particularly sensitive to several simulation parameters, and thus worse-performing.
Revised: We ﬁnd this baseline particularly sensitive to several simulation parameters, and thus worse-performing.

Original: In Table 2, we jointly estimate friction and elasticity parameters of our compliant contact model from video observations alone.
Revised: In Table 2, we jointly estimate friction and elasticity parameters of our compliant contact model from video observations alone.

Original: The trend is similar to Table 1, and ∇ Sim is able to precisely recover the parameters of the simulation.
Revised: The trend is similar to Table 1, and ∇ Sim is able to precisely recover the parameters of the simulation.

Original: A few examples can be seen in Fig.
Revised: A few examples can be seen in Fig.

Original: 4.1. D EFORMABLE B ODIES ( DEFORMABLE )
Revised: 4.1. D EFORMABLE B ODIES ( DEFORMABLE )

Original: We conduct a series of experiments to investigate the ability of ∇ Sim to recover physical parameters of deformable solids and thin-shell solids (cloth).
Revised: We conduct a series of experiments to investigate the ability of ∇ Sim to recover physical parameters of deformable solids and thin-shell solids (cloth).

Original: Our physical model is parameterized by the perparticle mass, and Lamé elasticity parameters, as described in in Appendix C.1.
Revised: Our physical model is parameterized by the perparticle mass, and Lamé elasticity parameters, as described in in Appendix C.1.

Original: Fig.
Revised: Fig.

Original: 3 illustrates the recovery of the elasticity parameters of a beam hanging under gravity by matching the deformation given by an input video sequence.
Revised: 3 illustrates the recovery of the elasticity parameters of a beam hanging under gravity by matching the deformation given by an input video sequence.

Original: We found our method is able to accurately recover the parameters of 100 instances of deformable objects (cloth, balls, beams) as reported in Table 3 and Fig.
Revised: We found our method is able to accurately recover the parameters of 100 instances of deformable objects (cloth, balls, beams) as reported in Table 3 and Fig.

Original: 4.1. S MOOTHNESS OF THE LOSS LANDSCAPE IN ∇ Sim
Revised: 4.1. S MOOTHNESS OF THE LOSS LANDSCAPE IN ∇ Sim

Original: Since ∇ Sim is a complex combination of differential non-linear components, we  verify whether meaningful gradients can be provided by analyzing the loss landscape.
Revised: Since ∇ Sim is a complex combination of differentiable non-linear components, we analyze the loss landscape to verify the validity of gradients through the system.

Original: Fig.
Revised: Fig.

Original: 4 depicts the loss landscape when optimizing for the mass of a rigid body whose shape, elastic, and frictional properties are known.
Revised: 4 illustrates the loss landscape when optimizing for the mass of a rigid body when all other physical properties are known.

Original: Wetake an object with unit mass ( 1 kg), and examine the behavior of mean-squared error over a range of initializations ( 0 .
Revised: We examine the image-space mean-squared error (MSE) of a unit-mass cube ( 1 kg) for a range of initializations ( 0 .

Original: 1 kg to 5 kg).
Revised: 1 kg to 5 kg).

Original: Notably, the loss landscape of ∇ Sim is well-behaved and conducive to momentum-based optimizers.
Revised: Notably, the loss landscape of ∇ Sim is well-behaved and conducive to momentum-based optimizers.

Original: Applying MSE to the ﬁrst and last frames of the predicted and true videos provides the best gradients for this task.
Revised: Applying MSE to the ﬁrst and last frames of the predicted and true videos provides the best gradients.

Original: However, for a naive gradient estimator applied to a non-differentiable simulator (PyBullet + REINFORCE),there are multiple local minima and the optimization is very sensitive to the initial guess.
Revised: However, for a naive gradient estimator applied to a nondifferentiable simulator (PyBullet + REINFORCE), multiple local minima exist resulting in a very narrow region of convergence.

Original: Thus, this indicates that ∇ Sim gradients provide useful information to estimate physical parameters, and justiﬁes ∇ Sim ’s superior performance in Tables 1, 2, 3.
Revised: This explains ∇ Sim ’s superior performance in Tables 1, 2, 3.

Original: 4. V ISUOMOTOR CONTROL
Revised: 4. V ISUOMOTOR CONTROL

Original: To investigate whether the gradients computed by ∇ Sim are meaningful for vision-based tasks, we conduct a range of visuomotor control experiments involving the actuation of deformable objects towards a visual target pose (a single image).
Revised: To investigate whether the gradients computed by ∇ Sim are meaningful for vision-based tasks, we conduct a range of visuomotor control experiments involving the actuation of deformable objects towards a visual target pose (a single image).

Original: In all cases, we evaluate against diffphysics , which uses a goal speciﬁcation and a reward, both deﬁned over the 3D state-space .
Revised: In all cases, we evaluate against diffphysics , which uses a goal speciﬁcation and a reward, both deﬁned over the 3D state-space .

Original: [Figure]
Revised: [Figure]

Original: 4.2. D EFORMABLE SOLIDS ( CONTROL - WALKER , CONTROL - FEM )
Revised: 4.2. D EFORMABLE SOLIDS ( CONTROL - WALKER , CONTROL - FEM )

Original: The ﬁrst example ( control-walker ) involves a 2D walker model.
Revised: The ﬁrst example ( control-walker ) involves a 2D walker model.

Original: Our goal is to train a neural network (NN) control policy to actuate the walker to reach a target pose on the right-hand side of an image.
Revised: Our goal is to train a neural network (NN) control policy to actuate the walker to reach a target pose on the right-hand side of an image.

Original: Our NN consists of one fully connected layer and a tanh activation.
Revised: Our NN consists of one fully connected layer and a tanh activation.

Original: The network input is a set of 8 time-varying sinusoidal signals and the output is a scalar activation value per-tetrahedron.
Revised: The network input is a set oftime-varying sinusoidal signals and the output is a scalar activation value per-tetrahedron.

Original: ∇ Sim is able to solve this environment within 3 iterations of gradient descent, by minimizing a pixelwise MSE between the last frame of the rendered video and the goal image as shown in Fig.
Revised: ∇ Sim is able to solve this environment within three iterations of gradient descent, by minimizing a pixelwise MSE between the last frame of the rendered video and the goal image as shown in Fig.

Original: 5 (column 2, bottom)
Revised: 5 (lower left).

Original: In our second test we formulate a more challenging 3D control problem ( control-fem ) where the goal is to actuate a soft-body FEM object (a gear ) consisting of 1152 tetrahedral elements to move to a target position as shown in Fig.
Revised: In our second test we formulate a more challenging 3D control problem ( control-fem ) where the goal is to actuate a soft-body FEM object (a gear ) consisting of 1152 tetrahedral elements to move to a target position as shown in Fig.

Original: 5 (left).
Revised: 5 (center).

Original: We use the same NN architecture as in the 2D walker example, and use the Adam (Kingma & Ba, 2015) optimizer to minimize a pixelwise MSE loss.
Revised: We use the same NN architecture as in the 2D walker example, and use the Adam (Kingma & Ba, 2015) optimizer to minimize a pixelwise MSE loss.

Original: We also train a privileged baseline ( diffphysics ) that uses strong supervision and minimizes the MSE between the target position and the precise 3D location of the center-of-mass (COM) of the FEM model at each time step (i.e.: a dense reward).
Revised: We also train a privileged baseline ( diffphysics ) that uses strong supervision and minimizes the MSE between the target position and the precise 3D location of the center-of-mass (COM) of the FEM model at each time step (i.e. a dense reward).

Original: We test both diffphysics and ∇ Sim against a naive baseline that generates random activations and plot convergence behaviors in Fig.
Revised: We test both diffphysics and ∇ Sim against a naive baseline that generates random activations and plot convergence behaviors in Fig.

Original: 6a.
Revised: 6a.

Original: While diffphysics appears to be a strong performer on this task, it is important to note that it uses explicit 3D supervision at each timestep (i.e., 30 fps).
Revised: While diffphysics appears to be a strong performer on this task, it is important to note that it uses explicit 3D supervision at each timestep (i.e. 30 FPS).

Original: In contrast, ∇ Sim uses a single image as an implicit target, and yet manages to achieve the goal state, albeit taking a longer number of iterations.
Revised: In contrast, ∇ Sim uses a single image as an implicit target, and yet manages to achieve the goal state, albeit taking a longer number of iterations.

Original: 4.2. C LOTH ( CONTROL - CLOTH )
Revised: 4.2. C LOTH ( CONTROL - CLOTH )

Original: We design an experiment to control a piece of cloth by optimizing the initial velocity such that it reaches a pre-speciﬁed target.
Revised: We design an experiment to control a piece of cloth by optimizing the initial velocity such that it reaches a pre-speciﬁed target.

Original: In each episode , a random cloth is spawned, comprising betweenand 2048 triangles, and a new start/goal combination is chosen.
Revised: In each episode , a random cloth is spawned, comprising betweenand 2048 triangles, and a new start/goal combination is chosen.

Original: In this challenging setup, we notice that state-based MPC ( diffphysics ) is often unable to accurately reach the target.
Revised: In this challenging setup, we notice that state-based MPC ( diffphysics ) is often unable to accurately reach the target.

Original: We believe this is due to the underdetermined nature of the problem, since, for objects such as cloth, the COM by itself does not uniquely determine the conﬁguration of the object.
Revised: We believe this is due to the underdetermined nature of the problem, since, for objects such as cloth, the COM by itself does not uniquely determine the conﬁguration of the object.

Original: Visuomotor control on the other hand, provides a more well-deﬁned problem.
Revised: Visuomotor control on the other hand, provides a more well-deﬁned problem.

Original: An illustration of the task is presented in Fig.
Revised: An illustration of the task is presented in Fig.

Original: 5 (column 3), and the convergence of the methods shown in Fig.
Revised: 5 (column 3), and the convergence of the methods shown in Fig.

Original: 6b.
Revised: 6b.

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: physics performs well

Original: [Figure]
Revised: [Figure]

Original: assumes strong 3D supervision

Original: [Figure]

Original: ∇

Original: [Figure]

Original: to solve

Original: [Figure]

Original: task

Original: [Figure]

Original: image the

Original: [Figure]

Original: 4. I MPACT OF IMPERFECT DYNAMICS AND RENDERING MODELS
Revised: 4. I MPACT OF IMPERFECT DYNAMICS AND RENDERING MODELS

Original: Being a white box method, the performance of ∇ Sim relies on the choice of dynamics and rendering models employed.
Revised: Being a white box method, the performance of ∇ Sim relies on the choice of dynamics and rendering models employed.

Original: An immediate question that arises is “ how would the performance of ∇ Sim be impacted (if at all) by such modeling choices. ”
Revised: An immediate question that arises is “ how would the performance of ∇ Sim be impacted (if at all) by such modeling choices. ”

Original: We conduct multiple experiments targeted at investigating modelling errors and summarize them in Table 4 (left).
Revised: We conduct multiple experiments targeted at investigating modelling errors and summarize them in Table 4 (left).

Original: We choose a dataset comprising 90 objects equally representing rigid, deformable, and cloth types.
Revised: We choose a dataset comprising 90 objects equally representing rigid, deformable, and cloth types.

Original: By not modeling speciﬁc dynamics and rendering phenomena, we create the following 5 variants of our simulator.
Revised: By not modeling speciﬁc dynamics and rendering phenomena, we create the following 5 variants of our simulator.

Original: Unmodeled friction : We model all collisions as being frictionless 2.
Revised: Unmodeled friction : We model all collisions as being frictionless.

Original: Unmodeled elasticity : We model all collisions as perfectly elastic 3.
Revised: Unmodeled elasticity : We model all collisions as perfectly elastic. 3.

Original: Rigid-as-deformable : All rigid objects in the dataset are modeled as deformable objects 4. Deformable-as-rigid : All deformable objects in the dataset are modeled as rigid objects 5.
Revised: Rigid-as-deformable : All rigid objects in the dataset are modeled as deformable objects. Deformable-as-rigid : All deformable objects in the dataset are modeled as rigid objects.

Original: Photorealistic render : We employ a photorealistic renderer—as opposed to ∇ Sim ’s differentiable rasterizers—in generating the target images
Revised: Photorealistic render : We employ a photorealistic renderer—as opposed to ∇ Sim ’s differentiable rasterizers—in generating the target images.

Original: In all cases, we evaluate the accuracy with which the mass of the target object is estimated from a target video sequencethat is devoid of modeling discrepancies.
Revised: In all cases, we evaluate the accuracy with which the mass of the target object is estimated from a target video sequence devoid of modeling discrepancies.

Original: In general, we observe that imperfect dynamics models (such as unmodeled friction and elasticity, or modeling a rigid object as deformable or vice-versa) have a more profound impact on parameter identiﬁcation compared to imperfect renderers.
Revised: In general, we observe that imperfect dynamics models (i.e. unmodeled friction and elasticity, or modeling a rigid object as deformable or vice-versa) have a more profound impact on parameter identiﬁcation compared to imperfect renderers.

Original: 4.3. U NMODELED DYNAMICS PHENOMENON
Revised: 4.3. U NMODELED DYNAMICS PHENOMENON

Original: From Table 4 (left), we observe a noticeable performance drop when dynamics effects go unmodeled.
Revised: From Table 4 (left), we observe a noticeable performance drop when dynamics effects go unmodeled.

Original: Expectedly, the repurcussions of incorrect object type modeling (Rigid-as-deformable, Deformableas-rigid) are more severe compared to unmodeled contact parameters (friction, elasticity).
Revised: Expectedly, the repurcussions of incorrect object type modeling (Rigid-as-deformable, Deformableas-rigid) are more severe compared to unmodeled contact parameters (friction, elasticity).

Original: Modeling a deformable body as a rigid body results in irrecoverable deformation parameters and has the most severe impact on the recovered parameter set.
Revised: Modeling a deformable body as a rigid body results in irrecoverable deformation parameters and has the most severe impact on the recovered parameter set.

Original: 4.3. U NMODELED RENDERING PHENOMENON
Revised: 4.3. U NMODELED RENDERING PHENOMENON

Original: We also independently investigate the impact of unmodeled rendering effects (assuming perfect dynamics).
Revised: We also independently investigate the impact of unmodeled rendering effects (assuming perfect dynamics).

Original: We indepenently render ground-truth images and object foreground masks from a photorealistic renderer Pharr et al.
Revised: We indepenently render ground-truth images and object foreground masks from a photorealistic renderer (Pharr et al., 2016).

Original: We use these photorealistic renderings for ground-truth and perform physical parameter estimation from video.
Revised: We use these photorealistic renderings for ground-truth and perform physical parameter estimation from video.

Original: We notice that the performance obtained under this setting is superior compared to ones with dynamics model imperfections.
Revised: We notice that the performance obtained under this setting is superior compared to ones with dynamics model imperfections.

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original: 4.3. I MPACT OF SHADING AND TEXTURE CUES
Revised: 4.3. I MPACT OF SHADING AND TEXTURE CUES

Original: Although our work does not attempt to bridge the reality gap, we show early prototypes to assess phenomena such as shading/texture.
Revised: Although our work does not attempt to bridge the reality gap, we show early prototypes to assess phenomena such as shading/texture.

Original: Fig.shows the accuracy over time for mass estimation from video.
Revised: Fig. 7 shows the accuracy over time for mass estimation from video.

Original: We evaluate three variants of the renderer - “ Only color ”, “ Shading ”, and “ Texture ”.
Revised: We evaluate three variants of the renderer - “ Only color ”, “ Shading ” and “ Texture ”.

Original: The “ Only color ” variant renders each mesh element in the same color regardless of the position and orientation of the light source.
Revised: The “ Only color ” variant renders each mesh element in the same color regardless of the position and orientation of the light source.

Original: The “ Shading ” variant implements a Phong shading model and can model specular and diffuse reﬂections.
Revised: The “ Shading ” variant implements a Phong shading model and can model specular and diffuse reﬂections.

Original: The “ Texture ” variant also applies a nonuniform texture sampled from ShapeNet Chang et al.
Revised: The “ Texture ” variant also applies a non-uniform texture sampled from ShapeNet (Chang et al., 2015).

Original: We notice that shading and texture cues signiﬁcantly improve convergence speed.
Revised: We notice that shading and texture cues signiﬁcantly improve convergence speed.

Original: This is expected, as vertex colors often have very little appearance cues inside the object boundaries, leading to poor correspondences between the rendered and ground-truth images.
Revised: This is expected, as vertex colors often have very little appearance cues inside the object boundaries, leading to poor correspondences between the rendered and ground-truth images.

Original: Furthermore, textures seem to offer slight improvements in convergence speed over shaded models, as highlighted by the inset (log scale) plot in Fig.
Revised: Furthermore, textures seem to offer slight improvements in convergence speed over shaded models, as highlighted by the inset (log scale) plot in Fig.

Original: In terms of texture cues, we observe that texture-mapped

Original: Lambertian and Phong models perform superior to their shaded variants, as highlighted by the inset (log scale) plot in Fig.

Original: 4.3. T IMING ANALYSIS
Revised: 4.3. T IMING ANALYSIS

Original: Table 4 (right) shows simulation rates for the forward and backward passes of each module.
Revised: Table 4 (right) shows simulation rates for the forward and backward passes of each module.

Original: We report forward and backward pass rates separately for the differentiable physics (DP) and the differentiable rendering (DR) modules.
Revised: We report forward and backward pass rates separately for the differentiable physics (DP) and the differentiable rendering (DR) modules.

Original: The time complexity of ∇ Sim is a function of the number of tetrahedrons and/or triangles.
Revised: The time complexity of ∇ Sim is a function of the number of tetrahedrons and/or triangles.

Original: We illustrate the arguably more complex case of deformable object simulation for varying numbers of tetrahedra (ranging from 100 to 10000 ).
Revised: We illustrate the arguably more complex case of deformable object simulation for varying numbers of tetrahedra (ranging from 100 to 10000 ).

Original: Even in the case oftetrahedra—enough to contruct complex mesh models of multiple moving objects— ∇ Sim enables faster-than-real-time simulation ( 1500 steps/second).
Revised: Even in the case of 10000 tetrahedra—enough to contruct complex mesh models of multiple moving objects— ∇ Sim enables faster-than-realtime simulation ( 1500 steps/second).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We presented ∇ Sim , a versatile differentiable simulator that enables system identiﬁcation from videos by differentiating through physical processes governing dyanmics and image formation.
Revised: We presented ∇ Sim , a versatile differentiable simulator that enables system identiﬁcation from videos by differentiating through physical processes governing dyanmics and image formation.

Original: We demonstrated the beneﬁts of such a holistic approach by estimating physical attributes for timeevolving scenes with complex dynamics and deformations, all from raw video observations.
Revised: We demonstrated the beneﬁts of such a holistic approach by estimating physical attributes for timeevolving scenes with complex dynamics and deformations, all from raw video observations.

Original: We also demonstrated the applicability of this efﬁcient and accurate estimation scheme on end-to-end visuomotor control tasks.
Revised: We also demonstrated the applicability of this efﬁcient and accurate estimation scheme on end-to-end visuomotor control tasks.

Original: The latter case highlights ∇ Sim ’s efﬁcient integration with PyTorch, facilitating interoperability with existing machine learning modules.
Revised: The latter case highlights ∇ Sim ’s efﬁcient integration with PyTorch, facilitating interoperability with existing machine learning modules.

Original: Interesting avenues for future work include extending our differentiable simulation to contact-rich motion, articulated bodies and higher-ﬁdelity physically-based renderers – doing so takes us closer to operating in the real-world.
Revised: Interesting avenues for future work include extending our differentiable simulation to contact-rich motion, articulated bodies and higher-ﬁdelity physically-based renderers – doing so takes us closer to operating in the real-world.

Revised: A CKNOWLEDGEMENTS

Revised: KM and LP thank the IVADO fundamental research project grant for funding.

Revised: FG thanks CIFAR for project funding under the Catalyst program.

Revised: FS and LP acknowledge partial support from NSERC.

Original: A A PPENDIX A D IFFERENTIABLE PHYSICS ENGINE
Revised: A A PPENDIX A D IFFERENTIABLE PHYSICS ENGINE

Original: Under Lagrangian mechanics, the state of a physical system can be described in terms of generalized coordinates q , generalized velocities ˙ q = u , and design, or model parameters θ .
Revised: Under Lagrangian mechanics, the state of a physical system can be described in terms of generalized coordinates q , generalized velocities ˙ q = u , and design, or model parameters θ .

Original: For the purposes of exposition, we make no distinction between rigid-bodies, deformable solids, or thin-shell models of cloth, etc.
Revised: For the purposes of exposition, we make no distinction between rigid-bodies, deformable solids, or thin-shell models of cloth and other bodies.

Original: Although the speciﬁc choices of coordinates and parameters vary, the simulation procedure is virtually unchanged.
Revised: Although the speciﬁc choices of coordinates and parameters vary, the simulation procedure is virtually unchanged.

Original: We denote the combined state vector by s ( t ) =
Revised: We denote the combined state vector by s ( t ) =

Original: [ q ( t ) , u ( t )] .
Revised: [ q ( t ) , u ( t )] .

Original: The dynamic evolution of the system is governed by a second order differential equations (ODE) of the form M ¨ s = f ( s ) , where M is a mass matrix that may also depend on our state and design parameters θ .
Revised: The dynamic evolution of the system is governed by a second order differential equations (ODE) of the form M ¨ s = f ( s ) , where M is a mass matrix that may also depend on our state and design parameters θ .

Original: Solutions to ODEs of this type may be obtained through black box numerical integration methods, and their derivatives calculated through the continuous adjoint method Chen et al.
Revised: Solutions to ODEs of this type may be obtained through black box numerical integration methods, and their derivatives calculated through the continuous adjoint method Chen et al.

Original: However, we instead consider our physics engine as a differentiable operation that provides an implicit relationship between a state vector s − =
Revised: However, we instead consider our physics engine as a differentiable operation that provides an implicit relationship between a state vector s − =

Original: s ( t ) at the start of a time step, and the updated state at the end of the time step s + = s ( t + ∆ t ) .
Revised: s ( t ) at the start of a time step, and the updated state at the end of the time step s + = s ( t + ∆ t ) .

Original: An arbitrary discrete time integration scheme can be then be abstracted as the function g ( s − , s + , θ ) = 0 , relating the initial and ﬁnal system state and the model parameters θ .
Revised: An arbitrary discrete time integration scheme can be then be abstracted as the function g ( s − , s + , θ ) = 0 , relating the initial and ﬁnal system state and the model parameters θ .

Original: By the implicit function theorem, if we can specify a loss function l at the output of the simulator, we can compute ∂l∂ s − as c T ∂ g ∂ s − , where c is the solution to the linear system ∂ g ∂
Revised: By the implicit function theorem, if we can specify a loss function l at the output of the simulator, we can compute ∂l∂ s − as c T ∂ g ∂ s − , where c is the solution to the linear system ∂ g ∂

Original: s + T c = − ∂l∂ s + T , and likewise for the model parameters θ .
Revised: s + T c = − ∂l∂ s + T , and likewise for the model parameters θ .

Original: While the partial derivatives ∂ g ∂ s − , ∂ g ∂ s + , ∂ g ∂θ can be computed by graph-based auto-differentation frameworks Paszke et al.
Revised: While the partial derivatives ∂ g ∂ s − , ∂ g ∂ s + , ∂ g ∂θ can be computed by graph-based automatic differentation frameworks Paszke et al.

Original: ; Abadi et al.
Revised: ; Abadi et al.

Original: (2015); Bradbury et al.
Revised: (2015); Bradbury et al.

Original: (2018), program transformation approaches such as DiffTaichi, and Google Tangent Hu et al.
Revised: (2018), program transformation approaches such as DiffTaichi, and Google Tangent Hu et al.

Original: (2020); van Merriënboer et al.
Revised: (2020); van Merriënboer et al.

Original: (2018) are particularly well-suited to simulation code.
Revised: (2018) are particularly well-suited to simulation code.

Original: We use an embedded subset of Python syntax, which computes the adjoint of each simulation kernel at runtime, and generates C++/CUDA Kirk et al.
Revised: We use an embedded subset of Python syntax, which computes the adjoint of each simulation kernel at runtime, and generates C++/CUDA Kirk et al.

Original: (2007) code.
Revised: (2007) code.

Original: Kernels are wrapped as custom autograd operations on PyTorch tensors, which allows users to focus on the deﬁnition of physical models, and leverage the PyTorch tape-based autodiff to track the overall program ﬂow.
Revised: Kernels are wrapped as custom autograd operations on PyTorch tensors, which allows users to focus on the deﬁnition of physical models, and leverage the PyTorch tape-based autodiff to track the overall program ﬂow.

Original: While this formulation is general enough to represent explicit, multi-step, or fully implicit time-integration schemes, we employ semi-implicit Euler integration, which is the preferred integration scheme for most simulators Erez et al.
Revised: While this formulation is general enough to represent explicit, multi-step, or fully implicit time-integration schemes, we employ semi-implicit Euler integration, which is the preferred integration scheme for most simulators Erez et al.

Original: A. P HYSICAL MODELS
Revised: A. P HYSICAL MODELS

Original: We now discuss some of the physical models available in ∇ Sim .
Revised: We now discuss some of the physical models available in ∇ Sim .

Original: Deformable Solids : As opposed to existing simulators that use grid-based methods for differentiable soft-body simulation Hu et al.
Revised: Deformable Solids : In contrast with existing simulators that use grid-based methods for differentiable soft-body simulation Hu et al.

Original: (2019; 2020), we adopt a ﬁnite element (FEM) model with constant strain tetrahedral elements common in computer graphics Sifakis & Barbic (2012).
Revised: (2019; 2020), we adopt a ﬁnite element (FEM) model with constant strain tetrahedral elements common in computer graphics Sifakis & Barbic (2012).

Original: We use the stable
Revised: We use the stable

Original: Neo-Hookean constitutive model of Smith et al.
Revised: Neo-Hookean constitutive model of Smith et al.

Original: Smith et al.
Revised: Smith et al.

Original: (2018a) that derives per-element forces from the following strain energy density:
Revised: (2018a) that derives per-element forces from the following strain energy density:

Original: [Equation]
Revised: [Equation]

Original: (1) where I C , J are invariants of strain, θ =
Revised: (1) where I C , J are invariants of strain, θ =

Original: [ µ, λ ] are the Lamé parameters, and α is a per-element actuation value that allows the element to expand and contract.
Revised: [ µ, λ ] are the Lamé parameters, and α is a per-element actuation value that allows the element to expand and contract.

Original: Numerically integrating the energy density over each tetrahedral mesh element with volume V e gives the total elastic potential energy, U ( q , θ )
Revised: Numerically integrating the energy density over each tetrahedral mesh element with volume V e gives the total elastic potential energy, U ( q , θ )

Original: = (cid:80) V e Ψ e .
Revised: = (cid:80) V e Ψ e .

Original: The forces due to this potential f e ( s , θ ) =
Revised: The forces due to this potential f e ( s , θ ) =

Original: −∇ q U ( q , θ ) , can computed analytically, and their gradients obtained using the adjoint method ( cf.
Revised: −∇ q U ( q , θ ) , can computed analytically, and their gradients obtained using the adjoint method ( cf.

Original: Section 3.1).
Revised: Section 3.1).

Original: Deformable Thin-Shells : To model thin-shells such as clothing, we use constant strain triangular elements embedded in 3D. The Neo-Hookean constitutive model above is applied to model in-plane elastic deformation, with the addition of a bending energy f b ( s , θ )
Revised: Deformable Thin-Shells : To model thin-shells such as clothing, we use constant strain triangular elements embedded in 3D. The Neo-Hookean constitutive model above is applied to model in-plane elastic deformation, with the addition of a bending energy f b ( s , θ )

Original: =
Revised: =

Original: k b sin ( φ 2 + α ) d , where k b is the bending stiffness, φ is the dihedral angle between two triangular faces, α is a per-edge actuation value that allows the mesh to ﬂex inwards or outwards, and d is the force direction given by Bridson et al.
Revised: k b sin ( φ 2 + α ) d , where k b is the bending stiffness, φ is the dihedral angle between two triangular faces, α is a per-edge actuation value that allows the mesh to ﬂex inwards or outwards, and d is the force direction given by Bridson et al.

Original: We also include a lift/drag model that approximates the effect of the surrounding air on the surface of mesh.
Revised: We also include a lift/drag model that approximates the effect of the surrounding air on the surface of mesh.

Original: Rigid Bodies : We represent the state of a 3D rigid body as q b =
Revised: Rigid Bodies : We represent the state of a 3D rigid body as q b =

Original: [ x , r ] consisting of a position x ∈ R 3 , and a quaternion r ∈ R 4 .
Revised: [ x , r ] consisting of a position x ∈ R 3 , and a quaternion r ∈ R 4 .

Original: The generalized velocity of a body is u b =
Revised: The generalized velocity of a body is u b =

Original: [ v , ω ] and the dynamics of each body is given by the Newton-Euler equations,
Revised: [ v , ω ] and the dynamics of each body is given by the Newton-Euler equations,

Original: [Equation]
Revised: [Equation]

Original: where the mass m and inertia matrix I (expressed at the center of mass) are considered design parameters θ .
Revised: where the mass m and inertia matrix I (expressed at the center of mass) are considered design parameters θ .

Original: Contact : We adopt a compliant contact model that associates elastic and damping forces with each nodal contact point.
Revised: Contact : We adopt a compliant contact model that associates elastic and damping forces with each nodal contact point.

Original: The model is parameterized by four scalars θ =
Revised: The model is parameterized by four scalars θ =

Original: [ k e , k d , k f , µ ] , corresponding to elastic stiffness, damping, frictional stiffness, and friction coefﬁcient respectively.
Revised: [ k e , k d , k f , µ ] , corresponding to elastic stiffness, damping, frictional stiffness, and friction coefﬁcient respectively.

Original: To prevent interpenetration we use a proportional penalty-based force, f n ( s , θ ) =
Revised: To prevent interpenetration we use a proportional penalty-based force, f n ( s , θ ) =

Original: − n [ k e C ( q )+ k d ˙ C ( u )] , where n is a contact normal, and C is a gap function measure of overlap projected on to R + .
Revised: − n [ k e C ( q )+ k d ˙ C ( u )] , where n is a contact normal, and C is a gap function measure of overlap projected on to R + .

Original: We model friction using a relaxed Coulomb model Todorov (2014)
Revised: We model friction using a relaxed Coulomb model Todorov (2014)

Original: f f ( s , θ ) =
Revised: f f ( s , θ ) =

Original: − D [ min ( µ | f n | , k f u s )] , where
Revised: − D [ min ( µ | f n | , k f u s )] , where

Original: D is a basis of the contact plane, and u s = D T u is the sliding velocity at the contact point.
Revised: D is a basis of the contact plane, and u s = D T u is the sliding velocity at the contact point.

Original: While these forces are only C 0 continuous, we found that this was sufﬁcient for optimization over a variety of objectives.
Revised: While these forces are only C 0 continuous, we found that this was sufﬁcient for optimization over a variety of objectives.

Original: More physical simulations : We also implement a number of other differentiable simulations such as pendula, mass-springs, and incompressible ﬂuids Stam (1999).
Revised: More physical simulations : We also implement a number of other differentiable simulations such as pendula, mass-springs, and incompressible ﬂuids Stam (1999).

Original: We note these systems have already been demonstrated in prior art, and thus focus on the more challenging systems in our paper.
Revised: We note these systems have already been demonstrated in prior art, and thus focus on the more challenging systems in our paper.

Original: B D ISCRETE A DJOINT M ETHOD
Revised: B D ISCRETE A DJOINT M ETHOD

Original: Above, we presented a formulation of time-integration using the discrete adjoint method that represents an arbitrary time-stepping scheme through the implicit relation,
Revised: Above, we presented a formulation of time-integration using the discrete adjoint method that represents an arbitrary time-stepping scheme through the implicit relation,

Original: [Equation]
Revised: [Equation]

Original: This formulation is general enough to represent both explicit or implicit time-stepping methods.
Revised: This formulation is general enough to represent both explicit or implicit time-stepping methods.

Original: While explicit methods are often simple to implement, they may require extremely small time-steps for stability, which is problematic for reverse-mode automatic differentiation frameworks that must explicitly store the input state for each discrete timestep invocation of the integration routine.
Revised: While explicit methods are often simple to implement, they may require extremely small time-steps for stability, which is problematic for reverse-mode automatic differentiation frameworks that must explicitly store the input state for each discrete timestep invocation of the integration routine.

Original: On the other hand, implicit methods can introduce computational overhead or unwanted numerical dissipation Hairer et al.
Revised: On the other hand, implicit methods can introduce computational overhead or unwanted numerical dissipation Hairer et al.

Original: For this reason, many real-time physics engines employ a semiimplicit ( symplectic ) Euler integration scheme Erez et al.
Revised: For this reason, many real-time physics engines employ a semiimplicit ( symplectic ) Euler integration scheme Erez et al.

Original: (2015), due to its ease of implementation and numerical stability in most meaningful scenarios (conserves energy for systems where the
Revised: (2015), due to its ease of implementation and numerical stability in most meaningful scenarios (conserves energy for systems where the

Original: Hamiltonian is time-invariant).
Revised: Hamiltonian is time-invariant).

Original: We now give a concrete example of the discrete adjoint method applied to semi-implicit Euler.
Revised: We now give a concrete example of the discrete adjoint method applied to semi-implicit Euler.

Original: For the state variables deﬁned above, the integration step may be written as follows,
Revised: For the state variables deﬁned above, the integration step may be written as follows,

Original: [Equation]
Revised: [Equation]

Original: Note that in general, the mass matrix M is a function of q and θ .
Revised: Note that in general, the mass matrix M is a function of q and θ .

Original: For conciseness we only consider the dependence on θ , although the overall procedure is unchanged in the general case.
Revised: For conciseness we only consider the dependence on θ , although the overall procedure is unchanged in the general case.

Original: We provide a brief sketch of computing the gradients of g ( s − , s + , θ ) .
Revised: We provide a brief sketch of computing the gradients of g ( s − , s + , θ ) .

Original: In the case of semi-implicit integration above, these are given by the following equations:
Revised: In the case of semi-implicit integration above, these are given by the following equations:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: In the case of semi-implicit Euler, the triangular structure of these Jacobians allows the adjoint variables to be computed explicitly.
Revised: In the case of semi-implicit Euler, the triangular structure of these Jacobians allows the adjoint variables to be computed explicitly.

Original: For fully implicit methods such as backwards Euler, the Jacobians may create a linear system that must be ﬁrst solved to generate adjoint variables.
Revised: For fully implicit methods such as backwards Euler, the Jacobians may create a linear system that must be ﬁrst solved to generate adjoint variables.

Original: C P HYSICAL M ODELS
Revised: C P HYSICAL M ODELS

Original: We now undertake a more detailed discussion of the physical models implemented in ∇ Sim .
Revised: We now undertake a more detailed discussion of the physical models implemented in ∇ Sim .

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: C. F INITE ELEMENT METHOD
Revised: C. F INITE ELEMENT METHOD

Original: As described in section 3.2 ("Physical models"), we use a hyperelastic constitutive model based on the neo-Hookean model of Smith et al.
Revised: As described in section 3.2 ("Physical models"), we use a hyperelastic constitutive model based on the neo-Hookean model of Smith et al.

Original: Smith et al.
Revised: Smith et al.

Original: (2018a):
Revised: (2018a):

Original: [Equation]
Revised: [Equation]

Original: The Lamé parameters, λ, µ , control the element’s resistance to shearing and volumetric strains.
Revised: The Lamé parameters, λ, µ , control the element’s resistance to shearing and volumetric strains.

Original: These may be speciﬁed on a per-element basis, allowing us to represent heterogeneous materials.
Revised: These may be speciﬁed on a per-element basis, allowing us to represent heterogeneous materials.

Original: In contrast to other work using particle-based models Hu et al.
Revised: In contrast to other work using particle-based models Hu et al.

Original: (2020), we adopt a mesh-based discretization for deformable shells and solids.
Revised: (2020), we adopt a mesh-based discretization for deformable shells and solids.

Original: For thin-shells, such as cloth, the surface is represented by a triangle mesh as in Figure 8a, enabling straightforward integration with our triangle mesh-based differentiable rasterizer Liu et al.
Revised: For thin-shells, such as cloth, the surface is represented by a triangle mesh as in Figure 8a, enabling straightforward integration with our triangle mesh-based differentiable rasterizer Liu et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: For solids, we use a tetrahedral FEM model as illustrated in Figure 8b.
Revised: For solids, we use a tetrahedral FEM model as illustrated in Figure 8b.

Original: Both these models include a per-element activation parameter α , which for thin-shells, allows us to control the relative dihedral angle between two connected faces.
Revised: Both these models include a per-element activation parameter α , which for thin-shells, allows us to control the relative dihedral angle between two connected faces.

Original: For tetrahedral meshes, this enables changing the element’s volume, enabling locomotion, as in the control-fem example. 
Revised: For tetrahedral meshes, this enables changing the element’s volume, enabling locomotion, as in the control-fem example. C [Equation]

Original: C.  C ONTACT
Revised:  C ONTACT

Original: Implicit contact methods based on linear complementarity formulations (LCP) of contact may be used to maintain hard non-penetration constraints de Avila Belbute-Peres et al.
Revised: Implicit contact methods based on linear complementarity formulations (LCP) of contact may be used to maintain hard non-penetration constraints de Avila Belbute-Peres et al.

Original: However, we found relaxed models of contact—used in typical physics engines Erez et al.
Revised: However, we found relaxed models of contact—used in typical physics engines Erez et al.

Original: (2015)—were sufﬁcient for our experiments.
Revised: (2015)—were sufﬁcient for our experiments.

Original: In this approach, contact forces are derived from a one-sided quadratic potential, giving rise to penalty forces of the form 9a.
Revised: In this approach, contact forces are derived from a one-sided quadratic potential, giving rise to penalty forces of the form 9a.

Original: While Coulomb friction may also be modeled as an LCP, we use a relaxed model where the stick regime is represented by a stiff quadratic potential around the origin, and a linear portion in the slip regime, as shown in Figure 9b.
Revised: While Coulomb friction may also be modeled as an LCP, we use a relaxed model where the stick regime is represented by a stiff quadratic potential around the origin, and a linear portion in the slip regime, as shown in Figure 9b.

Original: To generate contacts, we test each vertex of a mesh against a collision plane and introduce a contact within somethreshold distance  d .
Revised: To generate contacts, we test each vertex of a mesh against a collision plane and introduce a contact within some distance threshold d .

Original: C. P ENDULA

Original: We also implement simple and double pendula, as toy examples of well-behaved and chaotic systems respectively, and estimate the parameters of the system (i.e., the length(s) of the rod(s) and initial

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Penetration Depth - -0.

Original: F o r c e
Revised: F o r c e

Original: ExactRelaxed (a) - -0.
Revised: ExactRelaxed (a) - -0.

Original: 1 Slip Velocity - -0.
Revised: 1 Slip Velocity - -0.

Original: F o r c e
Revised: F o r c e

Original: CoulombRelaxed (b)  angular displacement(s)), by comparing the rendered videos (assuming uniformly random initial guesses) with the true videos.
Revised: CoulombRelaxed (b) C. P ENDULA We also implement simple and double pendula, as toy examples of well-behaved and chaotic systems respectively, and estimate the parameters of the system (i.e., the length(s) of the rod(s) and initial angular displacement(s)), by comparing the rendered videos (assuming uniformly random initial guesses) with the true videos.

Original: As pendula have extensively been studied in the context of differentiable physics simulation Degrave et al.
Revised: As pendula have extensively been studied in the context of differentiable physics simulation Degrave et al.

Original: (2016); de Avila Belbute-Peres et al.
Revised: (2016); de Avila Belbute-Peres et al.

Original: ; Cranmer et al.
Revised: ; Cranmer et al.

Original: (2020b);
Revised: (2020b);

Original: Toth et al.
Revised: Toth et al.

Original: ; Greydanus et al.
Revised: ; Greydanus et al.

Original: Sanchez-Gonzalez et al.
Revised: Sanchez-Gonzalez et al.

Original: (2019), we focus on more challenging systems which have not been studied in prior art.
Revised: (2019), we focus on more challenging systems which have not been studied in prior art.

Original: C. I NCOMPRESSIBLE FLUIDS
Revised: C. I NCOMPRESSIBLE FLUIDS

Original: As an example of incompressible ﬂuid simulation, we implement a smoke simulator following the popular semi-Lagrangian advection scheme of Stam et al.
Revised: As an example of incompressible ﬂuid simulation, we implement a smoke simulator following the popular semi-Lagrangian advection scheme of Stam et al.

Original: Stam (1999).
Revised: Stam (1999).

Original: At 2:20 in our supplementary video attachment, we show an experiment which optimizes the initial velocities of smoke particles to form a desired pattern.
Revised: At 2:20 in our supplementary video attachment, we show an experiment which optimizes the initial velocities of smoke particles to form a desired pattern.

Original: Similar schemes have already been realized differentiably, e.g. in
Revised: Similar schemes have already been realized differentiably, e.g. in

Original: DiffTaichi Hu et al.
Revised: DiffTaichi Hu et al.

Original: and autograd Maclaurin et al.
Revised: and autograd Maclaurin et al.

Original: D S OURCE - CODE TRANSFORMATION FOR AUTOMATIC DIFFERENTIATION
Revised: D S OURCE - CODE TRANSFORMATION FOR AUTOMATIC DIFFERENTIATION

Original: The discrete adjoint method requires computing gradients of physical quantities with respect to state and design parameters.
Revised: The discrete adjoint method requires computing gradients of physical quantities with respect to state and design parameters.

Original: To do so, we adopt a source code transformation approach to perform reverse mode automatic differentiation Hu et al.
Revised: To do so, we adopt a source code transformation approach to perform reverse mode automatic differentiation Hu et al.

Original: (2020); Margossian (2019).
Revised: (2020); Margossian (2019).

Original: We use a domain-speciﬁc subset of the Python syntax extended with primitves for representing vectors, matrices, and quaternions.
Revised: We use a domain-speciﬁc subset of the Python syntax extended with primitves for representing vectors, matrices, and quaternions.

Original: Each type includes functions for acting on them, and the corresponding adjoint method.
Revised: Each type includes functions for acting on them, and the corresponding adjoint method.

Original: An example simulation kernel is then deﬁned as follows:
Revised: An example simulation kernel is then deﬁned as follows:

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Revised: f

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Original:  Listing 1: Particle Integration Kernel
Revised: dt [Figure] [Figure] Listing 1: Particle Integration Kernel

Original: At runtime, the kernel’s abstract syntax tree (AST) is parsed using Python’s built-in ast module.
Revised: At runtime, the kernel’s abstract syntax tree (AST) is parsed using Python’s built-in ast module.

Original: We then generate C++ kernel code for forward and reverse mode, which may be compiled to a CPU or
Revised: We then generate C++ kernel code for forward and reverse mode, which may be compiled to a CPU or

Original: GPU executable using the PyTorch torch.utils.cpp_extension mechanism.
Revised: GPU executable using the PyTorch torch.utils.cpp_extension mechanism.

Original: This approach allows writing imperative code, with ﬁne-grained indexing and implicit operator fusion (since all operations in a kernel execute as one GPU kernel launch).
Revised: This approach allows writing imperative code, with ﬁne-grained indexing and implicit operator fusion (since all operations in a kernel execute as one GPU kernel launch).

Original: Each kernel is wrapped as a
Revised: Each kernel is wrapped as a

Original: PyTorch autograd operation so that it ﬁts natively into the larger computational graph.
Revised: PyTorch autograd operation so that it ﬁts natively into the larger computational graph.

Original: E MPC C ONTROLLER A RCHITECTURE
Revised: E MPC C ONTROLLER A RCHITECTURE

Original: For our model predictive control examples, we have used a simple 3-layer neural network architecture illustrated in Figure 10.
Revised: For our model predictive control examples, we use a simple 3-layer neural network architecture illustrated in Figure 10.

Original: With simulation time t as input we generate N phase-shifted sinusoidal signals which are passed to a fully-connected layer (zero-bias), and a ﬁnal activation layer.
Revised: With simulation time t as input we generate N phase-shifted sinusoidal signals which are passed to a fully-connected layer (zero-bias), and a ﬁnal activation layer.

Original: The output is a vector of per-element activation values as described in the previous section.
Revised: The output is a vector of per-element activation values as described in the previous section.

Original: F L OSS LANDSCAPES FOR PARAMETER ESTIMATION OF DEFORMABLE SOLIDS
Revised: F L OSS LANDSCAPES FOR PARAMETER ESTIMATION OF DEFORMABLE SOLIDS

Original: ∇ Sim integrates several functional blocks, many of which contain nonlinear operations.
Revised: ∇ Sim integrates several functional blocks, many of which contain nonlinear operations.

Original: Furthermore, we employ a pixelwise mean-squared error (MSE) loss function for estimating physical parameters from video.
Revised: Furthermore, we employ a pixelwise mean-squared error (MSE) loss function for estimating physical parameters from video.

Original: To demonstrate whether the gradients obtained from ∇ Sim are relevant for the task of physical parameter estimation, in Figure 2 of the main paper, we present an analysis of the MSE loss landscape for mass estimation.
Revised: To demonstrate whether the gradients obtained from ∇ Sim are relevant for the task of physical parameter estimation, in Figure 2 of the main paper, we present an analysis of the MSE loss landscape for mass estimation.

Original: F. E LASTICITY PARAMETER
Revised: F. E LASTICITY PARAMETER

Original: We now present a similar analysis for elasticity parameter estimation in deformable solids.
Revised: We now present a similar analysis for elasticity parameter estimation in deformable solids.

Original: Figure 11a shows the loss landscape when optimizing for the Lamé parameters of a deformable solid FEM.
Revised: Figure 11a shows the loss landscape when optimizing for the Lamé parameters of a deformable solid FEM.

Original: In this case, both parameters λ and µ are set to 1000 .
Revised: In this case, both parameters λ and µ are set to 1000 .

Original: As can be seen in the plot, the loss landscape has a unique, dominant minimum at 1000 .
Revised: As can be seen in the plot, the loss landscape has a unique, dominant minimum at 1000 .

Original: We believe the well-behaved nature of our loss landscape is a key contributing factor to the precise physical-parameter estimation ability of ∇ Sim .
Revised: We believe the well-behaved nature of our loss landscape is a key contributing factor to the precise physical-parameter estimation ability of ∇ Sim .

Original: F. L OSS LANDSCAPE IN P Y B ULLET (REINFORCE)
Revised: F. L OSS LANDSCAPE IN P Y B ULLET (REINFORCE)

Original: Figure 11 shows how optimization using REINFORCE can introduce complications.
Revised: Figure 11 shows how optimization using REINFORCE can introduce complications.

Original: As the simulation becomes unstable with masses close to zero, poor local optimum can arise near the mean of the current estimated mass.
Revised: As the simulation becomes unstable with masses close to zero, poor local optimum can arise near the mean of the current estimated mass.

Original: This illustrates that optimization through REINFORCE is only possible after careful tuning of step size, sampling noise and sampling range.
Revised: This illustrates that optimization through REINFORCE is only possible after careful tuning of step size, sampling noise and sampling range.

Original: This reduces the utility of this method in a realistic setting where these hyperparameters are not known a priori.
Revised: This reduces the utility of this method in a realistic setting where these hyperparameters are not known a priori.

Original: [Figure]
Revised: [Figure]

Original: F. I MPACT OF THE LENGTH OF A VIDEO SEQUENCE
Revised: F. I MPACT OF THE LENGTH OF A VIDEO SEQUENCE

Original: To assess the impact of the length of a video on the quality of our solution we plot the loss landscapes for videos of varying lengths in Fig.
Revised: To assess the impact of the length of a video on the quality of our solution, we plot the loss landscapes for videos of varying lengths in Fig.

Original: We ﬁnd that shorter videos tend to have steeper loss landscapes compared to longer ones.
Revised: We ﬁnd that shorter videos tend to have steeper loss landscapes compared to longer ones.

Original: The frame-rate also has an impact on the steepness of the landscape.
Revised: The frame-rate also has an impact on the steepness of the landscape.

Original: In all cases though, the loss landscape is smooth and has the same unique minimum.
Revised: In all cases though, the loss landscape is smooth and has the same unique minimum

Original:  G D ATASET DETAILS
Revised: . G D ATASET DETAILS

Original: For the rigid-body task of physical parameter estimation from video, we curated a dataset comprising of 14 meshes, as shown in Fig.
Revised: For the rigid-body task of physical parameter estimation from video, we curated a dataset comprising of 14 meshes, as shown in Fig.

Original: The objects include a combination of primitive shapes, fruits and vegetables, animals, ofﬁce objects, and airplanes.
Revised: The objects include a combination of primitive shapes, fruits and vegetables, animals, ofﬁce objects, and airplanes.

Original: For each of the experiments, we pick an object at random, and sample its physical attributes from a predeﬁned range: densities are in the range [2 , 12] kg/m 3 , contact parameters k e , k d , k f are in the range [1 , 500] , and the coefﬁcient of friction µ is in the range [0 . 2 , 1 . 0] .
Revised: For each experiment, we select an object at random, and sample its physical attributes from a predeﬁned range: densities from the range [2 , 12] kg/m 3 , contact parameters k e , k d , k f from the range [1 , 500] , and a coefﬁcient of friction µ from the range [0 . 2 , 1 . 0] .

Original: The positions, orientations, (anisotropic) scale factors, and initial velocities arerandomly uniformly sampled in a cube of side-length 13 m centered at the camera.
Revised: The positions, orientations, (anisotropic) scale factors, and initial velocities are sampled uniformly at random from a cube of side-length 13 m centered on the camera.

Original: Across all rigid-body experiments, we use 800 objects for training and 200 objects for testing.
Revised: Across all rigid-body experiments, we use 800 objects for training and 200 objects for testing.

Original: H B ASELINES
Revised: H B ASELINES

Original: In this section, we present implementation details of the baselineswe used in our experiments.
Revised: In this section, we present implementation details of the baselines used in our experiments.

Original: H. P Y B ULLET + REINFORCE
Revised: H. P Y B ULLET + REINFORCE

Original: To explore whether existing non-differentiable simulators can be employed for physical parameter estimation, we take PyBullet Coumans & Bai (2016–2019) – a popular physics engine – and make it trivially differentiable, by gradient estimation.
Revised: To explore whether existing non-differentiable simulators can be employed for physical parameter estimation, we take PyBullet Coumans & Bai (2016–2019) – a popular physics engine – and make it trivially differentiable, by gradient estimation.

Original: We employ the REINFORCE Williams (1992) technique to acquire an approximate gradient through the otherwise non-differentiable environment.
Revised: We employ the REINFORCE Williams (1992) technique to acquire an approximate gradient through the otherwise non-differentiable environment.

Original: The implementation was inspired by Wu et al.
Revised: The implementation was inspired by Wu et al.

Original: and Rezende et al.
Revised: and Rezende et al.

Original: In concurrent work, a similar idea was explored in Ehsani et al.
Revised: In concurrent work, a similar idea was explored in Ehsani et al.

Original: In PyBullet, the mass parameter of the object is randomly initialized in the range [0 , N v ] , where
Revised: In PyBullet, the mass parameter of the object is randomly initialized in the range [0 , N v ] , where

Original: N v is the number of vertices, the object is set to the same starting position and orientation as in the dataset, and the camera parameters are identical to those used in the dataset.
Revised: N v is the number of vertices, the object is set to the same starting position and orientation as in the dataset, and the camera parameters are identical to those used in the dataset.

Original: This conﬁguration ensures that if the mass was correct, the video frames rendered out by PyBullet would perfectly align with those generated by ∇ Sim .
Revised: This conﬁguration ensures that if the mass was correct, the video frames rendered out by PyBullet would perfectly align with those generated by ∇ Sim .

Original: Each episode is rolled out for the same duration as in the dataset ( 60 frames, corresponding to 2 seconds of motion).
Revised: Each episode is rolled out for the same duration as in the dataset ( 60 frames, corresponding to 2 seconds of motion).

Original: In PyBullet this is achieved by running the simulation at 240 Hz and skipping 7 frames between observations.
Revised: In PyBullet this is achieved by running the simulation at 240 Hz and skipping 7 frames between observations.

Original: The REINFORCE reward is calculated by summing the individual L 2 losses between ground truth frames and PyBullet frames, then multiplying each by − 1 to establish a global maximum at the correct mass, as opposed to a global minimum as in ∇ Sim .
Revised: The REINFORCE reward is calculated by summing the individual L 2 losses between ground truth frames and PyBullet frames, then multiplying each by − 1 to establish a global maximum at the correct mass, in contrast with a global minimum as in ∇ Sim .

Original: When all individual frame rewards have been calculated, all trajectory rewards are normalized before calculating the loss.
Revised: When all individual frame rewards have been calculated, all trajectory rewards are normalized before calculating the loss.

Original: This ensures that the reward is scaled correctly with respect to REINFORCE’s negative sample log likelihood, but when the mass value approaches the local optimum, this leads to instability in the optimization process.
Revised: This ensures that the reward is scaled correctly with respect to REINFORCE’s negative sample log likelihood, but when the mass value approaches the local optimum, this leads to instability in the optimization process.

Original: To mitigate this instability, we introduce reward decay, which a hyperparameter that slowly decreases the reward values as optimization progresses, in a similar manner to learning rate decay.
Revised: To mitigate this instability, we introduce reward decay, which a hyperparameter that slowly decreases the reward values as optimization progresses, in a similar manner to learning rate decay.

Original: Before each optimization step, all normalized frame reward values are multiplied by reward _ decay .
Revised: Before each optimization step, all normalized frame reward values are multiplied by reward _ decay .

Original: After the optimization step, the decay is updated by reward _ decay = reward _ decay ∗ decay _ factor .
Revised: After the optimization step, the decay is updated by reward _ decay = reward _ decay ∗ decay _ factor .

Original: The hyperparameters used in this baseline can be found in Table 5.Parameter
Revised: The hyperparameters used in this baseline can be found in Table 5.

Original: Value
Revised: [Table]

Original: Meaning no_samples How often was the mass sampledat every step optimization_steps
Revised: often was the mass sampled

Original: Total number of optimization steps sample_noise 0. Std. dev.
Revised: [Table]

Original: of normal distribution that mass is sampledfrom decay_factor 0.
Revised: normal distribution [Table] mass is sampled

Original: Factor that reward decay is multiplied with after optimizer step dataset_size
Revised: [Table]

Original: Number of bodies that the method was evaluated on
Revised: Number of bodies [Table] the method was evaluated [Table]

Original: H. CNN FOR DIRECT PARAMETER ESTIMATION
Revised: H. CNN FOR DIRECT PARAMETER ESTIMATION

Original: In the rigid-body parameter estimation experiments, we train a convnet baseline, building on the
Revised: In the rigid-body parameter estimation experiments, we train a ConvNet baseline, building on the

Original: EfﬁcientNet-B0 architecture Tan & Le (2019).
Revised: EfﬁcientNet-B0 architecture Tan & Le (2019).

Original: The convnet consists of 2 convolutional layers with parameters (PyTorch convention): (1280 , 128 , 1) , (128 , 32 , 1) , followed by linear layers and Relu activations with sizes [7680 , 1024 , 100 , 100 , 100 , 5] .
Revised: The ConvNet consists of two convolutional layers with parameters (PyTorch convention): (1280 , 128 , 1) , (128 , 32 , 1) , followed by linear layers and ReLU activations with sizes [7680 , 1024 , 100 , 100 , 100 , 5] .

Original: No activation is applied over the output of the convnet.
Revised: No activation is applied over the output of the ConvNet.

Original: We train the model to minimize the mean-squared error between the estimated and the true parameters, and use the Adam optimizer Kingma & Ba (2015) with learning rate of 0 . 0001 .
Revised: We train the model to minimize the mean-squared error between the estimated and the true parameters, and use the Adam optimizer Kingma & Ba (2015) with learning rate of 0 . 0001 .

Original: Each model was trained for 100 epochs on a V 100 GPU.
Revised: Each model was trained for 100 epochs on a V 100 GPU.

Original: The input image frames were preprocessed by resizing them to 64 × 64 pixels (to reduce GPU memory consumption) and the features were extracted with a pretrained EfﬁcientNet-B0.
Revised: The input image frames were preprocessed by resizing them to 64 × 64 pixels (to reduce GPU memory consumption) and the features were extracted with a pretrained EfﬁcientNet-B0.

Original: I C OMPUTE AND TIMING DETAILS
Revised: I C OMPUTE AND TIMING DETAILS

Original: Most of the models presented in ∇ Sim can be trained and evaluated on modern laptops equipped with graphics processing units (GPUs).
Revised: Most of the models presented in ∇ Sim can be trained and evaluated on modern laptops equipped with graphics processing units (GPUs).

Original: We ﬁnd that, on a laptop with an  i7 processor and aGeForce
Revised: We ﬁnd that, on a laptop with an Intel i7 processor and a

Original:  GTX 1060 GPU, parameter estimation experiments for rigid/nonrigid bodies can be run in under 5minutes per object on CPU and in under 1 minute on the GPU.
Revised: GeForce GTX 1060 GPU, parameter estimation experiments for rigid/nonrigid bodies can be run in under 5 - 20 minutes per object on CPU and in under 1 minute on the GPU.

Original: The visuomotor control experiments ( control-fem , control-cloth ) take about 30 minutes per episode on the CPU and underminutes per episode on the GPU.
Revised: The visuomotor control experiments ( control-fem , control-cloth ) take about 30 minutes per episode on the CPU and under 5 minutes per episode on the GPU.

Original: J O VERVIEW OF AVAILABLE DIFFERENTIABLE SIMULATIONS
Revised: J O VERVIEW OF AVAILABLE DIFFERENTIABLE SIMULATIONS

Original: Table 6 presents an overview of the differentiable simulations implemented in ∇ Sim , and the optimizable parameters therein.
Revised: Table 6 presents an overview of the differentiable simulations implemented in ∇ Sim , and the optimizable parameters therein.

Original: [Table]
Revised: [Table]

Original: K L IMITATIONS
Revised: K L IMITATIONS

Original: While providing a wide range of previously inaccessible capabilities, ∇ Sim has a few limitations that we discuss in this section.
Revised: While providing a wide range of previously inaccessible capabilities, ∇ Sim has a few limitations that we discuss in this section.

Original: These shortcomings also form interesting avenues for subsequent research.
Revised: These shortcomings also form interesting avenues for subsequent research.

Original: • ∇ Sim (and equivalently ∇ PyBullet) are inept at handling tiny masses ( 100 g and less).
Revised: • ∇ Sim (and equivalently ∇ PyBullet) are inept at handling tiny masses ( 100 g and less).

Original: Optimizing for physical parameters for such objects requires a closer look at the design of physics engine and possibly, numerical stability.
Revised: Optimizing for physical parameters for such objects requires a closer look at the design of physics engine and possibly, numerical stability.

Original: • Articulated bodies are not currently implemented in ∇ Sim .
Revised: • Articulated bodies are not currently implemented in ∇ Sim .

Original: Typically, articulated bodies are composed of multiple prismatic joints which lend additional degrees of freedom to the system.
Revised: Typically, articulated bodies are composed of multiple prismatic joints which lend additional degrees of freedom to the system.

Original: • While  contacts with simple geometries (such as between arbitrary triangle meshes and planar surfaces) are handled, ∇ Sim has limited capability to handle contact-rich motion that introduces a large number of discontinuities.
Revised: • While capable of modeling contacts with simple geometries (such as between arbitrary triangle meshes and planar surfaces), ∇ Sim has limited capability to handle contact-rich motion that introduces a large number of discontinuities.

Original: One way to handle contacts differentiably could be to employ more sophisticated contact detection techniques and solve a linear complementarity problem (LCP) at each step, as done in de Avila Belbute-Peres et al.
Revised: One way to handle contacts differentiably could be to employ more sophisticated contact detection techniques and solve a linear complementarity problem (LCP) at each step, as done in de Avila Belbute-Peres et al.

Original: • Aside from the aforementioned drawbacks, we note that physics engines are adept at modeling phenomena which can be codiﬁed.
Revised: • Aside from the aforementioned drawbacks, we note that physics engines are adept at modeling phenomena which can be codiﬁed.

Original: However, there are several unmodeled physical phenomena that occur in real-world videos and must be tackled in order for ∇ Sim to evolve as a scalable framework capable of operating in the wild.
Revised: However, there are several unmodeled physical phenomena that occur in real-world videos which must be studied in order for ∇ Sim to evolve as a scalable framework capable of operating in the wild.

Original: L B ROADER IMPACT
Revised: L B ROADER IMPACT

Original: Much progress has been made on end-to-end learning in visual domains.
Revised: Much progress has been made on end-to-end learning in visual domains.

Original: If successful, image and video understanding promises far-reaching applications from safer autonomous vehicles to more realistic computer graphics, but relying on these tools for planning and control poses substantial risk.
Revised: If successful, image and video understanding promises far-reaching applications from safer autonomous vehicles to more realistic computer graphics, but relying on these tools for planning and control poses substantial risk.

Original: Neural information processing systems have shown experimentally promising results on visuomotor tasks, yet fail in unpredictable and unintuitive ways when deployed in real-world applications.
Revised: Neural information processing systems have shown experimentally promising results on visuomotor tasks, yet fail in unpredictable and unintuitive ways when deployed in real-world applications.

Original: If embodied learning agents are to play a broader role in the physical world, they must be held to a higher standard of interpretability.
Revised: If embodied learning agents are to play a broader role in the physical world, they must be held to a higher standard of interpretability.

Original: Establishing trust requires not just empirical, but explanatory evidence in the form of physically grounded models.
Revised: Establishing trust requires not just empirical, but explanatory evidence in the form of physically grounded models.

Original: Our work provides a bridge between gradient- and model-based optimization.
Revised: Our work provides a bridge between gradient- and model-based optimization.

Original: Explicitly modeling visual dynamics using well-understood physical principles has important advantages for human explainability and debuggability.
Revised: Explicitly modeling visual dynamics using well-understood physical principles has important advantages for human explainability and debuggability.

Original: Unlike end-to-end neural architectures which distribute bias across a large set of parameters, ∇ Sim trades their ﬂexibility for physical interpretability.
Revised: Unlike end-to-end neural architectures which distribute bias across a large set of parameters, ∇ Sim trades their ﬂexibility for physical interpretability.

Original: This does not eliminate the risk of bias in simulation, but allows us to isolate bias to physically grounded variables.
Revised: This does not eliminate the risk of bias in simulation, but allows us to isolate bias to physically grounded variables.

Original: Where discrepancy occurs, users can probe the model to obtain end-to-end gradients with respect to variation in physical orientation and material properties, or pixelwise differences.
Revised: Where discrepancy occurs, users can probe the model to obtain end-to-end gradients with respect to variation in physical orientation and material properties, or pixelwise differences.

Original: Differentiable simulators like ∇ Sim afford a number of opportunities for use and abuse.
Revised: Differentiable simulators like ∇ Sim afford a number of opportunities for use and abuse.

Original: We envision the following scenarios.
Revised: We envision the following scenarios

Original: • A technician could query a trained model, "What physical parameters is the steering controller most sensitive to?", or "What happens if friction were slightly lower on that stretch of roadway?"
Revised: • A technician could query a trained model, "What physical parameters is the steering controller most sensitive to?", or "What happens if friction were slightly lower on that stretch of roadway?"

Original: • An energy-conscious organization could use ∇ Sim to accelerate convergence of reinforcement learning models, reducing the energy consumption required for training.
Revised: • An energy-conscious organization could use ∇ Sim to accelerate convergence of reinforcement learning models, reducing the energy consumption required for training.

Original: • Using differentiable simulation, an adversary could efﬁciently construct a physically plausible scene causing the model to produce an incorrect prediction or take an unsafe action.
Revised: • Using differentiable simulation, an adversary could efﬁciently construct a physically plausible scene causing the model to produce an incorrect prediction or take an unsafe action.

Original: Video understanding is a world-building exercise with inherent modeling bias.
Revised: Video understanding is a world-building exercise with inherent modeling bias.

Original: Using physically wellstudied models makes those modeling choices explicit, however mitigating the risk of bias still requires active human participation in the modeling process.
Revised: Using physically well-studied models makes those modeling choices explicit, however mitigating the risk of bias still requires active human participation in the modeling process.

Original: While a growing number of physically-based rendering and animation engines have been built, our approach requires a high upfront engineering cost in simulation infrastructure.
Revised: While a growing number of physically-based rendering and animation efforts are currently underway, our approach does require a high upfront engineering cost in simulation infrastructure.

Original: To operationalize these tools, we anticipate practitioners will need to devote signiﬁcant effort to identifying and replicating unmodeled dynamics from real worldtrajectories.
Revised: To operationalize these tools, we anticipate practitioners will need to devote signiﬁcant effort to identifying and replicating unmodeled dynamics from real world-trajectories.

Original: Differentiable simulation offers a computationally tractable and physically interpretable pathway for doing so, allowing users to estimate physical trajectories and the properties which govern them.
Revised: Differentiable simulation offers a computationally tractable and physically interpretable pathway for doing so, allowing users to estimate physical trajectories and the properties which govern them.