Original: VIRDO++: Real-World, Visuo-tactile Dynamics and Perception of Deformable Objects
Revised: VIRDO++: Real-World, Visuo-Tactile Dynamics and Perception of Deformable Objects

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract: Deformable objects manipulation can benefit from representations thatseamlessly integrate vision and touch while handling occlusions.
Revised: Abstract: Deformable objects manipulation can benefit from representations that seamlessly integrate vision and touch while handling occlusions.

Original: In this work, wepresent a novel approach for, and real-world demonstration of, multimodal visuotactile state-estimation and dynamics prediction for deformable objects.
Revised: In this work, we present a novel approach for, and real-world demonstration of, multimodal visuotactile state-estimation and dynamics prediction for deformable objects.

Original: Our approach, VIRDO++, builds on recent progress in multimodal neural implicit representations for deformable object state-estimation [1] via a new formulation fordeformation dynamics and a complementary state-estimation algorithm that (i)maintains a belief over deformations, and (ii) enables practical real-world application by removing the need forprivileged contact information.
Revised: Our approach, VIRDO++, builds on recent progress in multimodal neural implicit representations for deformable object state-estimation [1] via a new formulation for deformation dynamics and a complementary state-estimation algorithm that (i) maintains a belief distribution of deformation within a trajectory, and (ii) enables practical real-world application by removing the need for contact patches.

Original: In the context of tworeal-world robotic tasks, we show: (i) high-fidelity cross-modal state-estimationand prediction of deformable objects from partial visuo-tactile feedback, and (ii)generalization to unseen objects and contact formations. 
Revised: In the context of two real-world robotic tasks, we show: (i) high-fidelity cross-modal state-estimation and prediction of deformable objects from partial visuo-tactile feedback, and (ii) generalization to unseen objects and contact formations. [Figure]

Original: [Figure]
Revised: [Figure]

Original:  Introduction
Revised:  Introduction

Original: Deformable objects are ubiquitous in our everyday lives – the clothes we wear, the food we eat andmany of the tools we use are just a few examples.
Revised: Deformable objects are ubiquitous in our everyday lives – the clothes we wear, the food we eat, and many of the tools we use are just a few examples.

Original: As such, helpful robots of the future may benefitfrom the ability to master dexterous manipulation of deformable objects.
Revised: As such, helpful robots of the future may benefit from the ability to master dexterous manipulation of deformable objects.

Original: At the heart of this masteryis the interplay between object geometry and force transmission, perceived by vision and touch.
Revised: At the heart of this mastery is the interplay between object geometry and force transmission, perceived by vision and touch.

Original: Deformable object manipulation is difficult due to the complexity of this interplay (e g., infinitedimensional state-spaces and nonlinearity) and the ensuing challenges in perception (e g., partialobservability and occlusion) and controls [2, 3, 4, 5].
Revised: Deformable object manipulation is difficult due to the complexity of this interplay (e.g., infinite dimensional state-spaces and nonlinearity) and the ensuing challenges in perception (e.g., partial observability and occlusion) and controls [2, 3, 4, 5].

Original: Multimodal visuo-tactile representations can help address many of these challenges by exploiting mutual information and complementary cues
Revised: Multimodal visuo-tactile representations can help address many of these challenges by exploiting mutual information and complementary cues.

Original: Existing deformable object manipulation approaches typically use one modality (mostly vision) andrely on finite element/particle-based techniques [6, 7, 8, 9, 10, 11, 12, 13] or leverage deep learningfor visual affordance/latent dynamics learning [14, 15, 16, 17, 18, 19].
Revised: Existing deformable object manipulation approaches typically use one modality (mostly vision) and rely on finite element/particle-based techniques [6, 7, 8, 9, 10, 11, 12, 13] or leverage deep learning for visual affordance/latent dynamics learning [14, 15, 16, 17, 18, 19].

Original: The former methods typicallyrely on privileged knowledge (e.g., occluded or unknown boundary conditions) and stop at systemidentification, limiting their ability to refine the underlying physics model by learning from data.
Revised: The former methods typically rely on privileged knowledge (e.g., occluded or unknown boundary conditions) and stop at system identification, limiting their ability to refine the underlying physics model by learning from data.

Original: Latter methods lack i) tactile feedback which is far more informative of contact than vision, and ii)structured representations which can limit generalization and increase sample-complexity.
Revised: Latter methods lack i) tactile feedback which is far more informative of contact than vision, and ii) structured representations which can limit generalization and increase sample-complexity.

Original: Here, we build on recent work [1] that provides a framework for visuo-tactile state estimation usingmultimodal implicit neural representations.
Revised: Here, we build on recent work [1] that provides a framework for visuo-tactile state estimation using multimodal implicit neural representations.

Original: This approach offers a number of advantages includingmultimodal sensory fusion, direct integration with raw sensory feedback, and computational speed.
Revised: This approach offers a number of advantages including multimodal sensory fusion, direct integration with raw sensory feedback, and computational speed.

Original: However, [1] does not address 1) deformation predictions (dynamics) and 2) is not directly applicable to real-world robotics settings.
Revised: However, [1] does not address 1) deformation predictions (dynamics) and 2) is not directly applicable to real-world robotics settings.

Original: The first limitation is due to the lack of a dynamics module andan emphasis on static scenes.
Revised: The first limitation is due to the lack of a dynamics module and an emphasis on static scenes.

Original: The second limitation is due to the need for the contact patch whichare typically impractical to measure in real-world robotic settings.
Revised: The second limitation is due to the need for the contact patch which are typically impractical to measure in real-world robotic settings.

Original: The primary contribution of this work is (i) a novel approach to, and (ii) real-world demonstrationof, performing multi-modal visuo-tactile state estimation and dynamics prediction for deformableobjects.
Revised: The primary contribution of this work is (i) a novel approach to, and (ii) real-world demonstration of, performing multi-modal visuo-tactile state estimation and dynamics prediction for deformable objects.

Original: Fig.
Revised: Fig.

Original: 1 shows how our proposed approach uses partial views and tactile feedback for deformation prediction and state-estimation in real-world applications.
Revised: 1 shows how our proposed approach uses partial views and tactile feedback for deformation prediction and state-estimation in real-world applications.

Original: Our contributions are:
Revised: Our contributions are:

Original: Representation of deformable objects dynamics conditioned on actions: Multimodal neural 
Revised: Representation of deformable objects dynamics conditioned on actions: Multimodal neural implicit representation with

Original: implicit representation with action-conditioned dynamics to predict future deformations.
Revised: action conditioned dynamics to predict future deformations.

Original: To demon- strate the utility of the dynamics model, we introduce a state estimation algorithm based on particle filtering to maintain a belief distribution over object deformations.
Revised: To demonstrate the utility of the dynamics model, we introduce a state estimation algorithm based on particle filtering to maintain a belief distribution over object deformations

Original: Real-world demonstrations on two challenging tasks: Scraping with a spatula and chain manipulation.
Revised: Real-world demonstrations on two challenging tasks: Scraping with a spatula and chain manipulation.

Original: We demonstrate both 3D geometry estimation in heavy occlusion and action-conditioneddeformation/reaction wrench prediction.
Revised: We demonstrate both 3D geometry estimation in heavy occlusion and action-conditioned deformation/reaction wrench prediction.

Original: This is enabled by introducing a novel contact embeddingthat eliminates the need for explicit contact patch information used in [1].
Revised: This is enabled by introducing a novel contact latent vector that eliminates the need for explicit contact patch information used in [1].

Original: We use these tasks todemonstrate generalization to unseen objects and novel environments  .
Revised: We use these tasks to demonstrate generalization to unseen objects and novel environments  Related Work

Revised: Deformable object modeling.

Original: Methodology
Revised: Recent studies have attempted to address challenges in system identification and high computation costs in conventional continuum mechanics [6, 7, 8, 9, 10, 11, 12, 13]. To ease the burden of system identification, prior studies infer the physics models’ parameters based on high-fidelity physics engines [20, 21, 22] and simple force-deformation relationship (e.g. Hooke’s law) [23, 24]. The inherent limitations of these approaches are that i) performance is confined to the underlying physics model, often with strong assumptions on the objects (uniform density/elasticity) and the force-deformation relationship (linearity), and ii) access to privileged information such as occluded or unknown boundary conditions.

Revised: More recent approaches propose computationally efficient elastic object modelings with minimum system identification using, for example, a potential energy propagation [25] and geometric motion estimates [26].

Revised: However, these methods suffer from balancing between model approximation and computation cost as well as maintaining seamless integration with robotic sensing modalities (RGB-D cameras and F/T sensors).

Revised: In this study, we adopt a data-driven approach using neural networks [1, 27, 28].

Revised: Learning directly from observation, our approach does not require object parameters (Young’s modulus, Poisson’s ratio) or a strong assumption of object composition.

Revised: Even so, it can capture complex non-linear elastic behaviors and exhibit fast computation times due to the architecture of the neural networks.

Original: [Figure]
Revised: 3D geometric representation of deformable objects. Most recent studies have adopted discrete geometry representation for deformable objects with finite resolution such as meshes [7, 10, 29] and key points [15, 21], sometimes accompanied with graph neural networks [30, 28]. These approaches have been used largely for 1D and 2D objects (rope, cloth, top view of cylindrical objects). Among them, structured representations (e.g., Mesh or GNN) are advantageous at tracking connectivity, mainly used for deformable objects with large self-occlusions (e.g. folded cloth or knot) [31, 32].

Revised: On the other hand, we address general 3D geometry representations which are not confined to specific types of objects and can work with large occlusions (e.g., wok-scraping).

Revised: Our work builds on dense implicit geometric representation which include signed distances fields [1, 27, 33], occupancy [34], volume density [35], and dense descriptors [36].

Revised: These approaches have several advantages over their discrete counterparts in dealing with large occlusion, high resolution surface representation [37], parameterization of 3D geometries [38, 39], and are useful for downstream tasks (e.g., contact location detection similar to Sec. 4.5).

Revised: We extend these representations to real-world multimodal deformable object dynamics and perception by integrating touch and robot actions.

Original: Our approach, shown schematically in Fig.
Revised: Our approach, shown schematically in Fig.

Original: 2, is composed of a latent deformation dynamics model and animplicit dense geometric representation.
Revised: 2, is composed of a latent deformation dynamics model and an implicit dense geometric representation.

Original: Using Hidden Markov Models as an analogy, the former componentplays the role of the hidden state transition, and the latter is the observation model.
Revised: Using Hidden Markov Models as an analogy, the former component plays the role of the hidden state transition, and the latter is the observation model.

Original: We expand on the latent deformation dynamics in Sec.
Revised: We expand on the latent deformation dynamics in Sec.

Original: 2.1 and discuss the implicitgeometric representation in Sec. 2.2.
Revised: 3.1 and discuss the implicit geometric representation in Sec. 3.2.

Original: 2.1 Deformable Object Dynamics
Revised: 3.1 Deformable Object Dynamics

Original: VIRDO++ represents deformable object states using latent object ( α ) and force ( z t ) codes.
Revised: VIRDO++ represents deformable object states using latent object ( α ) and force ( z t ) codes.

Original: The object codecondenses the undeformed shape information into a feature representation that can extrapolate to new unseen object variants.
Revised: The object code condenses the undeformed shape information into a feature representation that can extrapolate to new unseen object variants.

Original: The force code is generated by the Force Module ( F ) which encodes boundary conditions as z t = F ( f t , c t , p t ) where f t ∈ R 6 is the reaction wrench at the wrist, p t ∈ R 6 is robot end-effector pose, and c t ∈ R
Revised: The force code is generated by the Force Module ( F ) which encodes boundary conditions as z t = F ( f t , c t , p t ) where f t ∈ R 6 is the reaction wrench at the wrist, p t ∈ R 6 is robot end-effector pose, and c t ∈ R

Original: lc is the contact embedding.
Revised: lc is the contact latent vector, where lc is the dimension of the contact latent vector.

Original: The contact embedding c t is a learnable latent vector that can change over time, whose primary roleis to store information about contact.
Revised: The contact latent vector c t is a learnable latent vector that can change over time whose primary role is to store contact information.

Original: However, it can also implicitly integrate information about theobject’s physical properties (e.g., stiffness) and historical information about the object’s states anddeformations.
Revised: However, it can also implicitly integrate information about the object’s physical properties (e.g., stiffness) and historical information about the object’s states and deformations.

Original: This embedding generalizes the original VIRDO [1] in terms of both (i) no longerneeding known contact patch locations, unlocking the capability to run VIRDO in the real-world,and (ii) representing additional useful information beyond just the contact location as was the case in VIRDO.
Revised: This embedding generalizes the original VIRDO [1] in terms of both (i) no longer needing known contact patch locations, unlocking the capability to run VIRDO in the real-world, and (ii) representing additional useful information beyond just the contact location as was the case in VIRDO.

Original: In more detail, VIRDO [1] used ground truth contact patches encoded by a PointNetencoder [20] directly which required access to this privileged information.
Revised: In more detail, VIRDO [1] used ground truth contact patches encoded by a PointNet encoder [40] directly which required access to this privileged information.

Original: Here, the contact patchand encoder are replaced by the learnable contact embedding.
Revised: Here, the contact patch and encoder are replaced by the learnable contact latent vector.

Original: The Action Module ( A ) predicts the next step’s boundary conditions given the current (latent) stateand robot action: A ( α , z t , a t )
Revised: The Action Module ( A ) predicts the next step’s boundary conditions given the current (latent) state and robot action: A ( α , z t , a t )

Original: =
Revised: =

Original: ˆ f t +1 , ˆ c t +1 where ˆ f t +1 ∈ R 6 is the predicted reaction wrench atthe wrist,
Revised: ˆ f t +1 , ˆ c t +1 where ˆ f t +1 ∈ R 6 is the predicted reaction wrench at the wrist,

Original: ˆ c t
Revised: ˆ c t

Original: +

Revised: +

Original: is the predicted contact embedding, and a t ∈ R 6 is the robot action (Cartesian displacement).
Revised: is the predicted contact latent vector, and a t ∈ R 6 is the robot action (Cartesian displacement).

Original: This predictive capability enables state-estimation and planning for deformable objectmanipulation.
Revised: This predictive capability enables state-estimation and downstream tasks such as contact location detection crucial for deformable object manipulation.

Revised: Though we did not include in this paper, the predictive ability of VIRDO++ also enables planning for deformable object manipulation.

Original: VIRDO lacked this functionality and could only infer the object geometry statically.
Revised: VIRDO lacked this functionality and could only infer the object geometry statically.

Original: 2.2 Geometry Representation
Revised: 3.2 Geometry Representation

Original: VIRDO++ decouples the 3D geometric representation of deformable objects into an undeformed(nominal) shape signed-distance field (SDF) and a set of deformation fields, similar to [1, 21].
Revised: VIRDO++ decouples the 3D geometric representation of deformable objects into an undeformed (nominal) shape signed-distance field (SDF) and a set of deformation fields, similar to [1, 39].

Original: Thedeformation field is a 3D vector field that, when summed with the deformed object SDF, results inthe nominal shape SDF.
Revised: The deformation field is a 3D vector field that, when summed with the deformed object SDF, results in the nominal shape SDF.

Original: More precisely, any query point x ∈ R 3 in the robot wrist frame belongingto the deformed shape SDF can be mapped back to the corresponding point x ′ of the nominal shape
Revised: More precisely, any query point x ∈ R 3 in the robot wrist frame belonging to the deformed shape SDF can be mapped back to the corresponding point x ′ of the nominal shape

Original: SDF by applying the point-wise deformation field ∆ x as x + ∆ x = x ′ .
Revised: SDF by applying the point-wise deformation field ∆ x as x + ∆ x = x ′ .

Original: In more detail:
Revised: In more detail:

Original: Nominal Shape Representation: The Object Module O ( x ′ ) is the parametric representation of thenominal shape SDF – the object shape with no external contacts corresponding to ∆ x = 0 .
Revised: Nominal Shape Representation: The Object Module O ( x ′ ) is the parametric representation of the nominal shape SDF – the object shape with no external contacts corresponding to ∆ x = 0 .

Original: Here, weuse a feedforward neural network to parameterize the SDF [22] and write O ( x ′ | Ψ
Revised: Here, we use a feedforward neural network to parameterize the SDF [38] and write O ( x ′ | Ψ

Original: o ( α ))
Revised: o ( α ))

Original: =
Revised: =

Original: s wheres is the signed-distance at query point x ′ and Ψ
Revised: s where s is the signed-distance at query point x ′ and Ψ

Original: o
Revised: o

Original: ( α ) is a hyper-network that predicts the weightsof O conditioned on the object code α .
Revised: ( α ) is a hyper-network that predicts the weights of O conditioned on the object code α .

Original: The object code and hyper-network weights are learnedend-to-end in an auto-decoder approach [22].
Revised: The object code and hyper-network weights are learned end-to-end in an auto-decoder approach [38].

Original: We simplify our notation to O ( x ′ ) for the remainder.
Revised: We simplify our notation to O ( x ′ ) for the remainder.

Original: Deformation Field Representation: The generalized geometric representation of a deformableobject is given by SDF ( x ) =
Revised: Deformation Field Representation: The generalized geometric representation of a deformable object is given by SDF ( x ) =

Original: O Φ o (cid:0) x
Revised: O Φ o (cid:0) x

Original: + D ( x | Ψ d ( z t , α t )) (cid:1)
Revised: + D ( x | Ψ d ( z t , α t )) (cid:1)

Original: =
Revised: =

Original: s .
Revised: s .

Original: Here, the Deformation Module
Revised: Here, the Deformation Module

Original: D ( x | Ψ d ( z t , α t )) =
Revised: D ( x | Ψ d ( z t , α t )) =

Original: ∆ x produces an object’s deformation field given force and object code pairs.
Revised: ∆ x produces an object’s deformation field given force and object code pairs.

Original: Similar to the Object Module, Deformation Module’s weights are given by the hyper-network Ψ d .
Revised: Similar to the Object Module, Deformation Module’s weights are given by the hyper-network Ψ d .

Original: 2.3 Training and Loss Formulation
Revised: 3.3 Training and Loss Formulation

Original: The training dataset for VIRDO++ is a set of trajectories S
Revised: The training dataset for VIRDO++ is a set of trajectories S

Original: = {T 1 , T 2 , ..., T N } , where each trajectory T j
Revised: = {T 1 , T 2 , ..., T N } , where each trajectory T j

Original: =
Revised: =

Original: ( p 0 , f 0 , P 0 , a 0 ) , ( p 1 , f 1 , P 1 , a 1 ) , ..., ( p k , f k , P k , a k ) } has k sequential observation and action tuples.
Revised: ( p 0 , f 0 , P 0 , a 0 ) , ( p 1 , f 1 , P 1 , a 1 ) , ..., ( p k , f k , P k , a k ) } has k sequential observation and action tuples.

Original: Here P j is a partial point cloud.
Revised: Here P j is a partial point cloud.

Original: Our first step is to pretrain a nominalshape representation (i.e., hyper-network Ψ o and object code α ).
Revised: Our first step is to pretrain a nominal shape representation (i.e., hyper-network Ψ o and object code α ).

Original: The loss for training m nominal shapes is L nominal =
Revised: The loss for training m nominal shapes is L nominal =

Original: L sdf + λ 1 L latent + λ 2 L hyper as in [1], where L latent = (cid:80) mi =1 ∥ α i ∥ 2 ,
Revised: L sdf + λ 1 L latent + λ 2 L hyper as in [1], where L latent = (cid:80) mi =1 ∥ α i ∥ 2 ,

Original: L hyper = (cid:80) mi =1 ∥
Revised: L hyper = (cid:80) mi =1 ∥

Original: Ψ o ( α i ) ∥ 2 , and:
Revised: Ψ o ( α i ) ∥ 2 , and:

Original: L sdf = m (cid:88) i = (cid:16) (cid:88) ¯ x ∈ Ω | clamp ( O i ( x ) , δ ) − clamp ( s ∗ , δ )
Revised: L sdf = m (cid:88) i = (cid:16) (cid:88) ¯ x ∈ Ω | clamp ( O i ( x ) , δ ) − clamp ( s ∗ , δ )

Original: | + λ (cid:88) ¯ x ∈
Revised: | + λ (cid:88) ¯ x ∈

Original: Ω(1 − ⟨∇
Revised: Ω(1 − ⟨∇

Original: O i ( x ) , n ∗ ⟩ ) (cid:17) .
Revised: O i ( x ) , n [Equation]

Original: n ∗ is the ground truth normal vector, δ is a parameter that clips SDF predictions, and s ∗ is theground truth signed distance.
Revised: n ∗ is the ground truth normal vector, δ is a parameter that clips SDF predictions, and s ∗ is the ground truth signed distance.

Original: After the pretraining, we train the rest of the modules using the threelosses L geo , L pred and L reg summed over a fixedtime horizon ( w ) while freezing Ψ o and the objectcode α :
Revised: After the pretraining, we train the rest of the modules using the three losses L geo , L pred and L reg summed over a fixed horizon ( w ) while freezing Ψ o and the object code α :

Original: [Equation]
Revised: [Equation]

Original: The geometry representation loss is defined as:
Revised: The geometry representation loss is defined as:

Original: [Equation]
Revised: [Equation]

Original: where x ′ = x + D Ψ d ( x ) , Ω is the 3D querying space, Ω 0 ⊂ Ω is the on-surface region, P :=
Revised: where x ′ = x + D Ψ d ( x ) , Ω is the 3D querying space, Ω 0 ⊂ Ω is the on-surface region, P :=

Original: { p | p ∈Ω 0 } is an unordered set of the on-surface points, and ¯ P ∗ is the ground truth nominal shape pointcloud.
Revised: { p | p ∈ Ω 0 } is an unordered set of the on-surface points, and ¯ P ∗ is the ground truth nominal shape point cloud.

Original: Next, we add a reaction wrench prediction loss as L pred =
Revised: Next, we add a reaction wrench prediction loss as L pred =

Original: ∥
Revised: ∥

Original: f t +1 − ˆ f t +1 ∥ .
Revised: f t +1 − ˆ f t +1 ∥ .

Original: The predictionloss for ˆ c t +1 is implicitly handled by Eq. 2 because the contact embedding prediction is recursivelyused as next step’s input during training.
Revised: The prediction loss for ˆ c t +1 is implicitly handled by Eq. 2 because the contact latent vector prediction is recursively used as next step’s input during training.

Original: In contrast, we do replace ˆ f t +1 with real measurementin Eq. 2.
Revised: In contrast, we do replace ˆ f t +1 with real measurement in Eq. 2.

Original: Finally, regularization losses L regt =
Revised: Finally, regularization losses L regt =

Original: λ 9 ∥
Revised: λ 9 ∥

Original: z t ∥ 2 + λ 10 ∥ c t ∥ 

Revised: z t ∥ 2 + λ 10 ∥ c t ∥

Original: + λ 11 ∥
Revised: + λ 11 ∥

Original: Ψ d ( α , z t ) ∥ 2 , whichregularize the force codes, contact embedding, and the weights of D respectively.
Revised: Ψ d ( α , z t ) ∥ 2 , which regularize the force codes, contact latent vector, and the weights of D respectively.

Original: 2.4 Inference and State-estimation Algorithm
Revised: 3.4 Inference and State-estimation Algorithm

Original: In order to maintain a belief distribution over the object deformation, we develop a particle filterbased inference algorithm, Alg. 1.
Revised: To maintain a belief distribution over the object deformation, we develop a particle filter-based inference algorithm, Alg. 1.

Original: Each particle represents a contact embedding which can be usedto reconstruct the full 3D object geometry.
Revised: Each particle represents a contact latent vector which can be used to reconstruct the full 3D object geometry.

Original: The Refine step updates the particle set by performinggradient descent on min ˆC t L geo ( P t , ˆC t , f t , p t ) with the visible object point cloud.
Revised: The Refine step updates the particle set by performing gradient descent on min ˆ C t L geo ( P t , ˆ C t , f t , p t ) with the visible object point cloud.

Original: The particlesare then weighted according to their wrench error prediction with Weight-Function defined asw it =
Revised: The particles are then weighted according to their wrench error prediction with Weight-Function defined as w it =

Original: exp ( − γ ( ˆ f it − f t )) and resampled with probability proportional to this weight vector.
Revised: exp ( − γ ( ˆ f it − f t )) and resampled with probability proportional to this weight vector.

Original: Here,ˆ f it is the reaction wrench prediction of the i th particle and f t is a ground truth measurement (see
Revised: Here, ˆ f it is the reaction wrench prediction of the i th particle and f t is a ground truth measurement.

Original: Appendix ). Finally, the particles are propagated through the Force and Action Modules, a smallamount of uncertainties is applied ∼ N (0 , 0 . 01) to the particles and the algorithm is repeated. 
Revised: Finally, the particles are propagated through the Force and Action Modules, a small amount of uncertainties is applied ∼ N (0 , 0 . 01) to the particles and the algorithm is repeated. (see algorithm variations in

Original:  1: procedure 2: [Equation]
Revised: Appendix Sec. A.2.2) 1: procedure 2: ˆC 0 ← { ˆ c 00 , ˆ c 10 , ..., ˆ c n − 1} ∼ N (0 , 0 . 01)

Original: ▷ Init Particles 3: for each scan in Trajectory Length do 4:
Revised: ▷ Init Particles 3: for each scan in Trajectory Length do 4:

Original: C t ← Refine ( P t , ˆC t , f t , p t ) 5:if scan is 0 then 6: ¯C t ← C t 7: else 8: w t ← Weight-Function ( f t , ˆ f 0 t , ˆ f 1 t , ..., ˆ f n − 1 t ) 9: ¯C t ← Re-sampling ( w t , C t ) 10: end if 11:
Revised: C t ← Refine ( P t , ˆC t , f t , p t ) 5: w t ← Weight-Function ( f t , ˆ f 0 t , ˆ f 1 t , ..., ˆ f n − 1 t ) 6: ¯C t ← Re-sampling ( w t , C t ) 7:

Original: Z t ← Force-Module ( ¯C t , f t , p t , α ) 12: { ( ˆ f 0 t +1 , ˆ c 0 t +1 ) , ( ˆ f 1 t +1 , ˆ c 1 t +1 ) , ..., ( ˆf n − 1 t
Revised: Z t ← Force-Module ( ¯C t , f t , p t , α ) 8: { ( ˆ f 0 t +1 , ˆ c 0 t +1 ) , ( ˆ f 1 t +1 , ˆ c 1 t +1 ) , ..., ( ˆf n − 1 t

Original: +1 , ˆ c n − 1 t +1 ) } ← Action-Module ( Z t , α , a t ) 13: t ← t +14: end for 15: end procedure
Revised: +1 , ˆ c n − 1 t +1 ) } ← Action-Module ( Z t , α , a t ) 9: t ← t +10: end for 11: end procedure

Original:  Experiments
Revised:  Experiments

Original: For real robot experiments, we investigate two classes of objects: spatulas and bike chains .
Revised: For real robot experiments, we investigate two classes of objects: spatulas and bike chains .

Original: Theseobjects exhibit different deformation behaviors: (i) spatulas are representative of bendable elasticobjects that retain their shape when external forces are removed, whereas (ii) bike chains (withthe 2 end points controlled by a bi-manual manipulator) can undergo significantly larger deformations (e.g., from gravity) – requiring algorithms that can reason over free-space while estimating incontact shapes and forces.
Revised: These objects exhibit different deformation behaviors: (i) spatulas are representative of bendable elastic objects that retain their shape when external forces are removed, whereas (ii) bike chains (with the 2 end points controlled by a bi-manual manipulator) can undergo significantly larger deformations (e.g., from gravity) – requiring algorithms that can reason over free-space while estimating incontact shapes and forces.

Original: Spatula deformations are generally local and experience significant reaction wrenches, while chain deformations are more global but with much smaller reaction wrenches.
Revised: Spatula deformations are generally local and experience significant reaction wrenches, while chain deformations are much larger but with much smaller reaction wrenches.

Original: 3.1 Spatula Manipulation 
Revised: 4.1 Spatula Manipulation Dataset

Original: [Figure]
Revised: [Figure]

Original: Dataset.

Original: Train and test objects are shown in Fig.
Revised: Train and test objects are shown in Fig.

Original: The training set consists of 4 spatulas, each with 672 samples ( N = 28 , k = 24 ), using 9:1train/test split (25 train and 3 test trajectories) for representation learning.
Revised: The training set consists ofspatulas, each with 672 samples ( N = 28 , k = 24 ), using 9:1 train/test split (25 train and 3 test trajectories) for representation learning.

Original: Thetest dataset consists of 2 held-out spatulas, each with 72 samples ( N = 3 ,k = 24 ), and is used to evaluate zero-shot transfer and not used for representation learning.
Revised: The test dataset consists of 2 held-out spatulas, each with 72 samples ( N = 3 , k = 24 ), and is used to evaluate zero-shot transfer and not used for representation learning.

Original: At the beginning of each trajectory ( k = 0 ), the endeffector moves to a random world Cartesian position in a bounding box ofsize (0 . 1 × 0 .
Revised: At the beginning of each trajectory ( k = 0 ), the end-effector moves to a random world Cartesian position in a bounding box of size (0 . 1 × 0 .

Original: 24 × 0 .
Revised: 24 × 0 .

Original: 06) meters and the tool is brought into contact with the table.
Revised: 06) meters and the tool is brought into contact with the table.

Original: Within a trajectory, the tool maintains contact while sampling a random action from( x, y, z, r, p, y )
Revised: Within a trajectory, the tool maintains contact while sampling a random action from ( x, y, z, r, p, y )

Original: =
Revised: =

Original: ± (0 . 02 , 0 . 02 , 0 . 01 , 0 . 05 , 0 . 04) where translations are in meters and rotations inradians.
Revised: ± (0 . 02 , 0 . 02 , 0 . 01 , 0 . 05 , 0 . 04) where translations are in meters and rotations in radians.

Original: For the hardware, we use Franka Emika’s Panda robot, ATI Gamma F/T sensor mounted onthe robot’s wrist and the Photoneo PhoXi 3D Scanner L. Wrench measurements are averaged over0.05 [s] and the point cloud is registered to the wrist frame. 
Revised: Wrench measurements are averaged over 0.05 [s] and the point cloud is registered to the wrist frame. The method only requires a full point cloud from the nominal geometry of the object. This is collected by holding the object in front of the camera and rotating. Deformed object geometries are collected from a single camera viewing the front side of the spatulas without occlusions and are all partial.

Original: P is then the result of segmenting theobject pointcloud and normalizing the scale into a bounding box of size 2 × 2 × 2[ m ] centered around(0 , 0 , 0) .
Revised: P is then the result of segmenting the object pointcloud and normalizing the scale into a bounding box of size 2 × 2 × 2[ m ] centered around (0 , 0 , 0) .

Original: The test dataset point clouds P contain occlusions at the bottom 0 .
Revised: The test dataset point clouds P contain occlusions at the bottom 0 .

Original: 15 × spatula-heightas shown in the top observation panel in Fig.
Revised: 15 × spatula-height as shown in the top observation panel in Fig.

Revised: For hardware, we use Franka Emika’s Panda, wrist mounted ATI Gamma F/T sensor, and Photoneo PhoXi 3D L.

Original: 3.1.1 State Estimation and Predictions
Revised: 4.2 State Estimation and Predictions

Original: CD (x1e3)

Original: Spatula w/ table

Original: Bike Chain

Original: [Table]
Revised: [Table]

Original: Here, we evaluate the deformation prediction(Sec. 2.1) and state-estimation (Alg. 1) accuracy of VIRDO++ on training and test trajectories.
Revised: Here, we evaluate the deformation prediction (Sec. 3.1) and state-estimation (Alg. 1) accuracy of VIRDO++ on training and test trajectories.

Original: Fig.
Revised: Fig.

Original: 4 shows an example of stateestimation for a test trajectory where the groundtruth (black) and reconstruction (magenta) areoverlaid.
Revised: 4 shows an example of stateestimation for a test trajectory where the ground truth (black) and reconstruction (magenta) are overlaid.

Original: We note the high  accuracyof stateestimation indicated by the overlay agreement, where the average distance between themeasured points and reconstructions is within0.03 [ m ] of normalized scale, just 1% of the object length.
Revised: We note the high state-estimation accuracy indicated by the overlay agreement, where the average distance between the measured points and reconstructions is within 0.03 [ m ] of normalized scale, just 1% of the object length.

Original: The quantitative results,reported in Tab. 1and Fig.
Revised: The quantitative results, Tab. 1, show

Original: 5, show that VIRDO++ is able to faithfully predict and estimatethe object deformation andreaction force despite significant occlusions.
Revised: VIRDO++ is able to faithfully predict and estimate object deformations and reaction forces despite significant occlusions.

Original: Force and torque norm errors are less than 0.567 [N],0.106{+[N], 0.
+} [Nm] (train) and 1.130 [N], 0.

Revised: Force and torque norm errors are less than 0.567 {+[N], 0.
+} [Nm] (train) and 1.130 [N], 0.

Original: [Nm] (test) respectively (seemore details in Appendix ).
Revised: [Nm] (test) respectively (see Appendix Sec. A.3.3 and Sec. A.3.4 for details).

Original: Our ablation study, Tab. 2(b), helps further anchor the scale of these state-estimation and dynamicsprediction errors.
Revised: Our ablation study, Tab. 3, helps further anchor the scale of these state-estimation and dynamics prediction errors.

Original: On state estimation, for example, our full VIRDO++ method achieves a Chamferdistance ( × 1 e 3 ) of normalized geometry is 0.93 [ m 2 ] .
Revised: On state estimation, for example, our full VIRDO++ method achieves a Chamfer distance ( × 10 3 ) of normalized geometry is 0.93 [ m 2 ] .

Original: The baseline model without a novel contactembedding input has a 12% increased Chamfer distance  as 1.06 [ m 2 ] .
Revised: The baseline model without a novel contact latent vector input has 27% increased chamfer distance error as 1.18 [ m 2 ] for geometry estimation.

Original:  Further if we assume theobject is rigid and reduce our state estimation to a “rigid pose estimation” pipeline, the Chamfer distance ( × 1 e 3 ) increases considerably to 16.19 [ m 2 ] .These ablations directly measure the quantitativevalue of VIRDO++’s contact embedding and deformation estimation and prediction.
Revised: Without c t , the baseline does not reason about contact during inference, resulting in larger errors during state-estimation Further if we assume the object is rigid and reduce our state estimation to a “rigid pose estimation” pipeline, the Chamfer distance ( × 10 3 ) increases considerably to 16.19 [ m 2 ] .

Original: We note that the geometry predictions are slightly less accurate than state-estimation. This is because the geometry prediction are computed by the learned dynamics using previous observationand action, while state-estimation uses the current measurement to update the prediction.
Revised: The geometry predictions are slightly less accurate than state-estimation because the geometry prediction are computed by the learned dynamics using previous observation and action, while stateestimation uses the current measurement to update the prediction.

Original: Despitethis, the prediction Chamfer distances are within 14% of the state estimation results.
Revised: Despite this, the prediction Chamfer distances are within 14% of the state estimation results.

Original: Moreover,the train and test trajectories results show 25% difference in Chamfer distance.
Revised: Moreover, the train and test trajectories results show 25% difference in Chamfer distance.

Original: This is interestingbecause the test trajectories are not only from unseen tool configurations but also with occlusions(see Dataset ).
Revised: This is interesting because the test trajectories are not only from unseen tool configurations but also with occlusions (see Dataset ).

Original: This suggests that learned dynamics of VIRDO++ generalize effectively to unseencontact formations with occlusions.t =
Revised: This suggests that learned dynamics of VIRDO++ generalize effectively to unseen contact formations with occlusions. [Equation]

Original: t
Revised: [Figure] [Equation] 4.3 Evaluation on a Downstream Manipulation Task - Extrinsic Contact Detection Central to compliant tool manipulation is estimating and controlling the contact between the tool and environment.

Original: =
Revised: For example, when scraping a wok with a spatula, the robot must reason over the contact formation (e.g., point, line, or patch) to ensure the spatula is scraping properly at the desired region on the wok. In this section, we evaluate VIRDO++ ability to estimate extrinsic contact features; specifically, the contact line created as the spatula is brought into contact with the table.

Original: t
Revised: To this end, we compute the signed distance values of the tables surface ( z = 0) to compute a contact cluster. This requires only a single feed-forward path where the input is an N × 3 matrix, where N is the number of query points. From the contact cluster, we extract a contact line by selecting the two points furthest apart similar to [41, 42, 43]. The evaluation metrics are the average Euclidean distance 12 Σ 2 i

Original: ={+=
+}
Revised: {+=
+}

Original: t =t = 
Revised: || l i − l ∗ i || , where l i is the detected contact lines, l ∗ i is the ground truth contact lines, and i = 0 , 1 refers to right and left ends of each contact line respectively. Ground truth contact lines were obtained from ground truth pointcloud without occlusions by applying a contact mask z ≤ ϵ . This contact feature estimation task is particularly interesting as it is the inverse problem of [1]; estimating [1]’s privileged input (=contact location) by inferring c t .

Original: t =t =
Revised: As shown in Tab. 2, VIRDO++ can accurately detect contact locations with less than 8 mm error on a flat table for the current and the following time step regardless of occlusions at the tip. As in Sec.4.2, test trajectories from 4 spatulas with 15% occlusion were used for the evaluation. Tab. shows that the average contact line detection errors from all 4 spatulas are less than 7 [mm] during estimation and are less than 8 [mm] for predictions in Euclidean error.

Revised: Similar to the result in

Original: [Figure]
Revised: [Table]

Original: G.
Revised: [Equation] [Table] [Equation]

Original: [Figure]
Revised: [Table]

Original: 3.2 Generalizations To Unseen Objects{+Sec. 4.2, estimations are 
+}
Revised: {+Sec. 4.2, estimations are 
+}

Original: [Figure]
Revised: [mm] more precise than predictions because they are optimized using the sensor measurements. Considering that the total length of the spatulas are about 200 [mm], this is less than 4% of error in total length. Estimating extrinsic contacts is a challenging task for rigid object [44] and is further complicated by object compliance; however, our approach proves effective due to its high-quality 3D reconstructions (details of error distribution in Appendix Fig. 12). Tab.also quantifies performance against ablations (without c t and Rigid-body assumption) illustrating the importance of the contact embedding and reasoning over compliance.

Revised: 4.4 Generalizations Tasks

Original: This section demonstrates VIRDO++ generalization ability to unseen objects and their dynamics.
Revised: This section demonstrates VIRDO++ generalization ability to unseen objects and their dynamics.

Original: For each unseen object, the generalization dataset contains 1 nominal point cloud and 3 trajectories,each with 24 transitions with occlusion.
Revised: For each unseen object, the generalization dataset contains 1 nominal point cloud and 3 trajectories, each with 24 transitions with occlusion.

Original: Here, we directly apply the same pre-trained model from
Revised: Here, we directly apply the same pre-trained model from

Original: Sec.
Revised: Sec.

Original: 3.1 to the generalization test dataset.
Revised: 4.1 to the generalization test dataset.

Original: 3.2.1 Object code inference 
Revised: 4.4.1 Object code inference for Unseen Objects

Original: [Figure]
Revised: Train Objects

Original:  VIRDO++ is able to represent nominal geometries unseen during training within distribution.
Revised: Test Objects VIRDO++ is able to represent nominal geometries unseen during training within distribution.

Original: Fig.
Revised: Fig.

Original: 6 shows reconstructions ofboth seen and unseen object in Fig.
Revised: 6 shows reconstructions of both seen and unseen object in Fig.

Original: 3 using a single pretrained
Revised: 3 using a single pretrained

Original: Object Module.
Revised: Object Module.

Original: For the unseen objects, we start from a randomlyinitialized object code α ∼ N (0 , 0 . 01) and update the code bygradient descent with Adam.
Revised: For the unseen objects, we start from a randomly initialized object code α ∼ N (0 , 0 . 01) and update the code by gradient descent with Adam.

Original: We use the same loss function from
Revised: We use the same loss function from

Original: Eq. 1 with pre-trained VIRDO++ while freezing the weights ofthe network.
Revised: Eq. 1 with pre-trained VIRDO++ while freezing the weights of the network.

Original: The first row of Tab. 2a shows the Chamfer distance error of the unseen objects alongside training objects, indicating the relatively small drop inperformance.
Revised: The first row of Tab. 3 shows the Chamfer distance error of the unseen objects alongside training objects, indicating the relatively small drop in performance.

Original: When comparing Fig.
Revised: When comparing Fig.

Original: 6 and the ground truth in Fig.
Revised: 6 and the ground truth in Fig.

Original: 3, we note that the test objectreconstructions are close to the ground truth aside from some fine details at the edges.
Revised: 3, we note that the test reconstructions are close to the ground truth aside from some fine edge details. t = t =t =t = t =t =

Original: 3.2.2 Zero Shot Dynamics Prediction and State Estimation
Revised: 4.4.2 Zero Shot Dynamics Prediction and State-estimation Generalization

Original: With the object codes obtained in Sec. 
Revised: With the object codes obtained in Sec. 4.4.

Original: 3.2.1, VIRDO++ is able to predict the unseen object’s dynamics based on their nominal shape.
Revised: [Table] VIRDO++ is able to predict the unseen object’s dynamics based on their nominal shape.

Original: Here, we evaluate both prediction and state-estimation performance as in Sec.
Revised: Here [Table] we evaluate both prediction and state-estimation performance as in Sec.

Original: We note that the observations of the unseen objects include occlusions asdiscussed in Dataset .
Revised: We note that the observations of the unseen objects include occlusions as discussed in Dataset .

Original: We present the results of this evaluation in Tab.2a and use Fig. 7 to provideintuition for Chamfer distance values.
Revised: Tab.

Original: The results show comparable performance between trainingand test objects with a small drop in accuracy, highlighting our model’s generalization capability.
Revised: 3 shows comparable performance between training and test objects with a small drop in accuracy, highlighting our model’s generalization capability.

Original: [Figure]
Revised: Sec.

Original: 3.3 Wok Scraping Application
Revised: A.3.6 in Appendix provides intuition for the Chamfer distance values.

Original: In this section, we show VIRDO++ generalization to novel environments – a wok with high curvatures producing greater variances in contact formations.
Revised: Further, we evaluate VIRDO++ ’s ability to perform high-fidelity geometry prediction and estimation even in novel environment (Wok scraping) without 3D model.

Original: Here, we add a training dataset with 2scraping trajectories on the wok and perform state estimation and predictions given a test trajectorywith 13 transitions. As in Fig. 8, VIRDO++ provides high-fidelity dense geometry of deformable
Revised: Details can be found in Appendix Sec. A.

Original: [Table]
Revised: [Table]

Original: objects in high occlusion cases without a 3D model or configuration of the environment; however, due to the complex environment, it occasionally produces undesired artifacts (2 out of 13 trans

Original: [Figure]

Original: 3.4 Bike Chain Application
Revised: 4.5 Bike Chain Application

Original: In this experiment, the chain ends are grasped by two arms.
Revised: In this experiment the chain ends are grasped by two arms.

Original: One end of the chain is moved randomly through the vertices of an equally-spaced 6 × 6 × 6 grid fitting inside of a 0 .
Revised: One end of the chain is moved randomly through the vertices of an equally-spaced 6 × 6 × 6 grid fitting inside of a 0 .

Original: 1 × 0 .
Revised: 1 × 0 .

Original: 18 × 0 . 2meter bounding box.
Revised: 18 × 0 .meter bounding box.

Original: The visited vertices of this grid make up the training dataset.
Revised: The visited vertices of this grid make up the training dataset.

Original: The test datasetis generated by sampling 20 random positions uniformly from the bounding box.
Revised: The test dataset is generated by sampling 20 random positions uniformly from the bounding box.

Original: Unlike spatulamanipulation, the end-effector orientations were fixed during bike chain manipulation because geometry changes are dominantly caused by translation rather than rotation; therefore, p t , a t ∈ R 3 .
Revised: The end-effector orientations were fixed; therefore, p t , a t ∈ R 3 .

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 9, VIRDO++ is able to predict/estimate both free-space and in-contact chain geometry.
Revised: 7, VIRDO++ is able to predict/estimate both free-space and in-contact chain geometry.

Original: The in-contact phase occurs when the arm moves sufficiently far down such that the chainlays on the table.
Revised: The in-contact phase occurs when the arm moves sufficiently far down such that the chain lays on the table.

Original: We report the quantitative results of geometry estimation and dynamics predictionin Tab. 1 and Fig.
Revised: We report the quantitative results of geometry estimation and dynamics prediction in Tab. 1 and Appendix Sec.

Original: The qualitative results are shown in Fig.
Revised: A.3.3.

Original: The empirical results are comparable[Equation] [Figure] tothe spatula experiments despitethe significantly larger shape variations.
Revised: The empirical results are comparable to spatula experiments despite significantly larger shape variations.

Original: Details on wrench prediction errorsare inthe Appendix , where the standard deviation in error is approximately an orderof magnitude smaller than the spatula. We hypothesize that this because i) the ground truth wrenchscale is an order of magnitude smaller than that of the spatula and ii) the action space dimension ishalf the one of the spatula resulting in a simpler model to learn.  Related Work Deformable object modeling.
Revised: Details on wrench prediction errors in Appendix Sec.

Original: Recent studies have tried to address challenges in system identification and high computation costs in conventional continuum mechanics [6, 7, 8, 9, 10, 11, 12, 13].

Original: Toease the burden of system identification, prior studies infer the physics models’ parameters based onhigh-fidelity physics engines [23, 24, 25] and simple force-deformation relationship (e.g. Hooke’slaw) [26, 27].

Original: The inherent limitations of these approaches are that i) performance is confined to theunderlying physics model, often with strong assumptions on the objects (uniform density/elasticity)and the force-deformation relationship (linearity), and ii) access to privileged information such asoccluded or unknown boundary conditions.

Original: More recent approaches propose computationally efficient elastic object modelings with minimum system identification using, for example, a potentialenergy propagation [28] and geometric motion estimates [29].

Original: However, these methods suffer frombalancing between model approximation and computation cost as well as maintaining seamless integration with robotic sensing modalities (RGB-D cameras and F/T sensors). In this study, we adopta data-driven approach using neural networks [1, 30, 31]. Learning directly from observation, ourapproach does not require object parameters (Young’s modulus, Poisson’s ratio) or a strong assumption of object composition. Even so, it can capture complex non-linear elastic behaviors and exhibitfast computation times due to the architecture of the neural networks.3D geometric representation of deformable objects.
Revised: A.3.4.

Original:  Most recent studies have adopted discretegeometry representation for deformable objects with finite resolution such as meshes [7, 10, 32] andkey points [15, 24], sometimes accompanied with graph neural networks [33, 31]. These approacheshave been used largely for 1D and 2D objects (rope, cloth, top view of cylindrical objects) with minimum occlusion. On the other hand, we address general 3D geometry representations, not confinedto specific types of objects, which can work with large occlusions (e.g., wok-scraping). Our workbuilds on dense implicit geometric representation which include signed distances fields [1, 30, 34],occupancy [35], volume density [36], and dense descriptors [37].
Revised:  Limitations

Original: These approaches have severaladvantages over their discrete counterparts in dealing with large occlusion, high resolution surfacerepresentation [38], parameterization of 3D geometries [22, 21], and are useful for downstream tasks(e.g.

Original: , contact location detection similar to Sec. 3.4).

Original: We extend these representations to real-worldmultimodal deformable object dynamics and perception by integrating touch and robot actions.

Original: Discussion and Limitations

Original: VIRDO++ provides a novel approach to multimodal deformation prediction using deformation fieldsconditioned on action, contact, and tactile/force feedback.

Original: Our proposed approach to combining anexplicit multi-modal deformation field with an implicit geometric representation can have broaderapplication to other implicit approaches (e.g. Neural Radiance Fields and Dense Descriptors).

Original: Whileour results with VIRDO++ are promising, its limitations give rise to future work: 1) Non-rigid grasps: VIRDO++ assumes that the gripper is rigidly attached to a fixed set of pointson the object.
Revised: Our proposed approach has a number of limitations: 1) Non-rigid grasps: VIRDO++ assumes that the gripper is rigidly holding the object.

Original: When it comes to non-rigid grasping, we need to estimate relative motion of the objects w.r.t the gripper and the force transmission is now a function of this relative motion.
Revised: When it comes to non-rigid grasping, we need to estimate relative motion of the objects w.r.t the gripper and the force transmission is now a function of this relative motion.

Original: Onepotential solution is to use collocated tactile sensor feedback [39, 40, 41] instead of simple F/Tsensing, which enables reasoning over reaction forces conditioned on the grasp.
Revised: One potential solution is to use collocated tactile sensor feedback [45, 46, 47] with F/T sensing to reason over reaction forces conditioned on the grasp.

Original:  Inertial ef- fects: our approach assumes quasi-static manipulation (negligible accelerations).
Revised: 2) Inertial effects: our approach assumes quasi-static manipulation (negligible accelerations).

Original: In terms of fast motion, the tools’ inertial effects may also impact the wrench measurements.
Revised: During fast motions, the tools’ inertial effects may also impact deformations and wrench measurements.

Original: In such cases, wesuggest supplementing VIRDO++ ’s input withadditional term such as end-effector accelerations.3)
Revised: When applicable, we suggest supplementing VIRDO++ ’s input with end-effector accelerations.

Original: Effective generalization across different environments: Although we provide an example ofenvironment generalization (wok experimentsshown in Fig. 8), the performance is not as stable asour other demonstrated applications.
Revised: 3) Generalization to novel environments: While we provide an example of environment generalization (wok experiments Fig. 14), the performance is lower than our other demonstrated applications.

Original: VIRDO++ may benefit from acomplete or partial 3D modelof the environment to integrate inter-penetration losses [27] or collision checkers [42].
Revised: VIRDO++ may benefit from a 3D model of the environment to integrate inter-penetration losses [24] or collision checkers [48]. Acknowledgments

Revised: This work was in part supported by Robotics at Google under the Google Faculty Research Award 2021 and the National Science Foundation (NSF) grant NRI-2220876.

Revised: Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation