Original: VIRDO++: Real-World, Visuo-Tactile Dynamics and Perception of Deformable Objects
Revised: VIRDO++: Real-World, Visuo-Tactile Dynamics and Perception of Deformable Objects

Original: Abstract: Deformable objects manipulation can benefit from representations that seamlessly integrate vision and touch while handling occlusions.
Revised: Abstract: Deformable objects manipulation can benefit from representations that seamlessly integrate vision and touch while handling occlusions.

Original: In this work, we present a novel approach for, and real-world demonstration of, multimodal visuotactile state-estimation and dynamics prediction for deformable objects.
Revised: In this work, we present a novel approach for, and real-world demonstration of, multimodal visuotactile state-estimation and dynamics prediction for deformable objects.

Original: Our approach, VIRDO++, builds on recent progress in multimodal neural implicit representations for deformable object state-estimation [1] via a new formulation for deformation dynamics and a complementary state-estimation algorithm that (i) maintains a belief distribution of deformation within a trajectory, and (ii) enables practical real-world application by removing the need for contact patches.
Revised: Our approach, VIRDO++, builds on recent progress in multimodal neural implicit representations for deformable object state-estimation [1] via a new formulation for deformation dynamics and a complementary state-estimation algorithm that (i) maintains a belief distribution of deformation within a trajectory, and (ii) enables practical real-world application by removing the need for contact patches.

Original: In the context of two real-world robotic tasks, we show: (i) high-fidelity cross-modal state-estimation and prediction of deformable objects from partial visuo-tactile feedback, and (ii) generalization to unseen objects and contact formations.[Figure] [Figure]
Revised: In the context of two real-world robotic tasks, we show: (i) high-fidelity cross-modal state-estimation and prediction of deformable objects from partial visuo-tactile feedback, and (ii) generalization to unseen objects and contact formations.

Original:  Introduction
Revised:  Introduction

Original: Deformable objects are ubiquitous in our everyday lives – the clothes we wear, the food we eat, and many of the tools we use are just a few examples.
Revised: Deformable objects are ubiquitous in our everyday lives – the clothes we wear, the food we eat and many of the tools we use are just a few examples.

Original: As such, helpful robots of the future may benefit from the ability to master dexterous manipulation of deformable objects.
Revised: As such, helpful robots of the future may benefit from the ability to master dexterous manipulation of deformable objects.

Original: At the heart of this mastery is the interplay between object geometry and force transmission, perceived by vision and touch.
Revised: At the heart of this mastery is the interplay between object geometry and force transmission, perceived by vision and touch.

Original: Deformable object manipulation is difficult due to the complexity of this interplay (e.g., infinite dimensional state-spaces and nonlinearity) and the ensuing challenges in perception (e.g., partial observability and occlusion) and controls [2, 3, 4, 5].
Revised: Deformable object manipulation is difficult due to the complexity of this interplay (e.g., infinite dimensional state-spaces and nonlinearity) and the ensuing challenges in perception (e.g., partial observability and occlusion) and controls [2, 3, 4, 5].

Original: Multimodal visuo-tactile representations can help address many of these challenges by exploiting mutual information and complementary cues.
Revised: Multimodal visuo-tactile representations can help address many of these challenges by exploiting mutual information and complementary cues.

Original: Existing deformable object manipulation approaches typically use one modality (mostly vision) and rely on finite element/particle-based techniques [6, 7, 8, 9, 10, 11, 12, 13] or leverage deep learning for visual affordance/latent dynamics learning [14, 15, 16, 17, 18, 19].
Revised: Existing deformable object manipulation approaches typically use one modality (mostly vision) and rely on finite element/particle-based techniques [6, 7, 8, 9, 10, 11, 12, 13] or leverage deep learning for visual affordance/latent dynamics learning [14, 15, 16, 17, 18, 19].

Original: The former methods typically rely on privileged knowledge (e.g., occluded or unknown boundary conditions) and stop at system identification, limiting their ability to refine the underlying physics model by learning from data.
Revised: The former methods typically rely on privileged knowledge (e.g., occluded or unknown boundary conditions) and stop at system identification, limiting their ability to refine the underlying physics model by learning from data.

Original: Latter methods lack i) tactile feedback which is far more informative of contact than vision, and ii) structured representations which can limit generalization and increase sample-complexity.
Revised: Latter methods lack (i) tactile feedback which is far more informative of contact than vision, and (ii) structured representations which can limit generalization and increase sample-complexity.

Original: Here, we build on recent work [1] that provides a framework for visuo-tactile state estimation using multimodal implicit neural representations.
Revised: Here, we build on recent work [1] that provides a framework for visuo-tactile state estimation using multimodal implicit neural representations.

Original: This approach offers a number of advantages including multimodal sensory fusion, direct integration with raw sensory feedback, and computational speed.
Revised: This approach offers a number of advantages including multimodal sensory fusion, direct integration with raw sensory feedback, and computational speed.

Original: However, [1] does not address 1) deformation predictions (dynamics) and 2) is not directly applicable to real-world robotics settings.
Revised: However, [1] does not address (i) deformation predictions (dynamics) and (ii) is not directly applicable to real-world robotics settings.

Original: The first limitation is due to the lack of a dynamics module and an emphasis on static scenes.
Revised: The first limitation is due to the lack of a dynamics module and an emphasis on static scenes.

Original: The second limitation is due to the need for the contact patch which are typically impractical to measure in real-world robotic settings.
Revised: The second limitation is due to the need for the contact patch which are typically impractical to measure in real-world robotic settings.

Original: The primary contribution of this work is (i) a novel approach to, and (ii) real-world demonstration of, performing multi-modal visuo-tactile state estimation and dynamics prediction for deformable objects.
Revised: The primary contribution of this work is (i) a novel approach to, and (ii) real-world demonstration of, performing multi-modal visuo-tactile state estimation and dynamics prediction for deformable objects.

Original: Fig.
Revised: Fig.

Original: 1 shows how our proposed approach uses partial views and tactile feedback for deformation prediction and state-estimation in real-world applications.
Revised: 1 shows how our proposed approach uses partial views and tactile feedback for deformation prediction and state-estimation in real-world applications.

Original: Our contributions are:
Revised: Our contributions are:

Original: Representation of deformable objects dynamics conditioned on actions: Multimodal neural implicit representation with 
Revised: Representation of deformable objects dynamics conditioned on actions: Multimodal neural implicit representation with action-conditioned dynamics to predict future deformations

Original: action conditioned dynamics 
Revised: To demonstrate the utility of the dynamics model, we introduce a state estimation algorithm based on particle filtering

Original: to
Revised: to

Original: predict future deformations. To demonstrate the utility of the dynamics model, we introduce a state estimation algorithm based on particle filtering to maintain a belief distribution over object deformations
Revised: maintain a belief distribution over object deformations

Original: Real-world demonstrations on two challenging tasks: Scraping with a spatula and chain manipulation.
Revised: Real-world demonstrations on two challenging tasks: Scraping with a spatula and chain manipulation.

Original: We demonstrate both 3D geometry estimation in heavy occlusion and action-conditioned deformation/reaction wrench prediction.
Revised: We demonstrate both 3D geometry estimation in heavy occlusion and action-conditioned deformation/reaction wrench prediction.

Original: This is enabled by introducing a novel contact latent vector that eliminates the need for explicit contact patch information used in [1].
Revised: This is enabled by introducing a novel contact latent vector that eliminates the need for explicit contact patch information used in [1].

Original: We use these tasks to demonstrate generalization to unseen objects and novel environments
Revised: We use these tasks to demonstrate generalization to unseen objects and novel environments

Original:  Related Work
Revised:  Related Work

Original: Deformable object modeling.
Revised: Deformable object modeling.

Original: Recent studies have attempted to address challenges in system identification and high computation costs in conventional continuum mechanics [6, 7, 8, 9, 10, 11, 12, 13].
Revised: Recent studies have attempted to address challenges in system identification and high computation costs in conventional continuum mechanics [6, 7, 8, 9, 10, 11, 12, 13].

Original: To ease the burden of system identification, prior studies infer the physics models’ parameters based on high-fidelity physics engines [20, 21, 22] and simple force-deformation relationship (e.g.
Revised: To ease the burden of system identification, prior studies infer the physics models’ parameters based on high-fidelity physics engines [20, 21, 22] and simple force-deformation relationship (e.g.

Original: Hooke’s law)[23, 24].
Revised: Hooke’s law)

Original: The inherent limitations of these approaches are that i) performance is confined to the underlying physics model, often with strong assumptions on the objects (uniform density/elasticity) and the force-deformation relationship (linearity), and ii) access to privileged information such as occluded or unknown boundary conditions.
Revised: The inherent limitations of these approaches are that (i) performance is confined to the underlying physics model, often with strong assumptions on the objects (uniform density/elasticity) and the force-deformation relationship (linearity), and (ii) access to privileged information such as occluded or unknown boundary conditions.

Original: More recent approaches propose computationally efficient elastic object modelings with minimum system identification using, for example, a potential energy propagation [25] and geometric motion estimates [26].
Revised: More recent approaches propose computationally efficient elastic object modelings with minimum system identification using, for example, a potential energy propagation [24] and geometric motion estimates [25].

Original: However, these methods suffer from balancing between model approximation and computation cost as well as maintaining seamless integration with robotic sensing modalities (RGB-D cameras and F/T sensors).
Revised: However, these methods suffer from balancing between model approximation and computation cost as well as maintaining seamless integration with robotic sensing modalities (RGB-D cameras and F/T sensors).

Original: In this study, we adopt a data-driven approach using neural networks [1, 27, 28].
Revised: In this study, we adopt a data-driven approach using neural networks [1, 26, 27].

Original: Learning directly from observation, our approach does not require object parameters (Young’s modulus, Poisson’s ratio) or a strong assumption of object composition.
Revised: Learning directly from observation, our approach does not require object parameters (Young’s modulus, Poisson’s ratio) or a strong assumption of object composition.

Original: Even so, it can capture complex non-linear elastic behaviors and exhibit fast computation times due to the architecture of the neural networks.
Revised: Even so, it can capture complex non-linear elastic behaviors and exhibit fast computation times due to the architecture of the neural networks.

Original: 3D geometric representation of deformable objects.
Revised: 3D geometric representation of deformable objects.

Original: Most recent studies have adopted discrete geometry representation for deformable objects with finite resolution such as meshes [7, 10, 29] and key points [15, 21], sometimes accompanied with graph neural networks [30, 28].
Revised: Most recent studies have adopted discrete geometry representation for deformable objects with finite resolution such as meshes [7, 10, 28] and key points [15, 21], sometimes accompanied with graph neural networks [29, 27].

Original: These approaches have been used largely for 1D and 2D objects (rope, cloth, top view of cylindrical objects).
Revised: These approaches have been used largely for 1D and 2D objects (rope, cloth, top view of cylindrical objects).

Original: Among them, structured representations (e.g., Mesh or GNN) are advantageous at tracking connectivity, mainly used for deformable objects with large self-occlusions (e.g. folded cloth or knot) [31, 32].
Revised: Among them, structured representations (e.g., Mesh or GNN) are advantageous at tracking connectivity, mainly used for deformable objects with large self-occlusions (e.g. folded cloth or knot) [30, 31].

Original: On the other hand, we address general 3D geometry representations which are not confined to specific types of objects and can work with large occlusions (e.g., wok-scraping).
Revised: On the other hand, we address general 3D geometry representations which are not confined to specific types of objects and can work with large occlusions (e.g., wok-scraping).

Original: Our work builds on dense implicit geometric representation which include signed distances fields [1, 27, 33], occupancy [34], volume density [35], and dense descriptors [36].
Revised: Our work builds on dense implicit geometric representation which include signed distances fields [1, 26, 32], occupancy [33], volume density [34], and dense descriptors [35].

Original: These approaches have several advantages over their discrete counterparts in dealing with large occlusion, high resolution surface representation [37], parameterization of 3D geometries [38, 39], and are useful for downstream tasks (e.g., contact location detection similar to Sec. 4.5).
Revised: These approaches have several advantages over their discrete counterparts in dealing with large occlusion, high resolution surface representation [36], parameterization of 3D geometries [37, 38], and are useful for downstream tasks (e.g., contact location detection similar to Sec. 4.5).

Original: We extend these representations to real-world multimodal deformable object dynamics and perception by integrating touch and robot actions.
Revised: We extend these representations to real-world multimodal deformable object dynamics and perception by integrating touch and robot actions.

Original:  Methodology
Revised:  Methodology

Original: Our approach, shown schematically in Fig.
Revised: Our approach, shown schematically in Fig.

Original: 2, is composed of a latent deformation dynamics model and an implicit dense geometric representation.
Revised: 2, is composed of a latent deformation dynamics model and an implicit dense geometric representation.

Original: Using Hidden Markov Models as an analogy, the former component plays the role of the hidden state transition, and the latter is the observation model.
Revised: Using Hidden Markov Models as an analogy, the former component plays the role of the hidden state transition, and the latter is the observation model.

Original: We expand on the latent deformation dynamics in Sec.
Revised: We expand on the latent deformation dynamics in Sec.

Original: 3.1 and discuss the implicit geometric representation in Sec. 3.2.
Revised: 3.1 and discuss the implicit geometric representation in Sec. 3.2.

Original: 3.1 Deformable Object Dynamics
Revised: 3.1 Deformable Object Dynamics

Original: VIRDO++ represents deformable object states using latent object ( α ) and force ( z t ) codes.
Revised: VIRDO++ represents deformable object states using latent object ( α ) and force ( z t ) codes.

Original: The object code condenses the undeformed shape information into a feature representation that can extrapolate tonew unseen object variants.
Revised: The object code condenses the undeformed shape information into a feature representation that can interpolate to unseen object variants.

Original: The force code is generated by the Force Module ( F ) which encodes boundary conditions as z t = F ( f t , c t , p t ) where f t ∈ R 6 is the reaction wrench at the wrist, p t ∈ R 6 is robot end-effector pose, and c t ∈ R
Revised: The force code is generated by the Force Module ( F ) which encodes boundary conditions as z t = F ( f t , c t , p t ) where f t ∈ R 6 is the reaction wrench at the wrist, p t ∈ R 6 is robot end-effector pose, and c t ∈ R

Original: lc is the contact latent vector, where lc is the dimension of the contact latent vector.
Revised: lc is the contact latent vector, where lc is the dimension of the contact latent vector.

Original: The contact latent vector c t is a learnable latent vector that can change over time whose primary role is to store contact information.
Revised: The contact latent vector c t is a learnable latent vector that can change over time whose primary role is to store contact information.

Original: However, it can also implicitly integrate information about the object’s physical properties (e.g., stiffness) and historical information about the object’s states and deformations.
Revised: However, it can also implicitly integrate information about the object’s physical properties (e.g., stiffness) and historical information about the object’s states and deformations.

Original: This embedding generalizes the original
Revised: This embedding generalizes the original

Original: VIRDO [1] in terms of both (i) no longer needing known contact patch locations, unlocking the capability to run VIRDO in the real-world, and (ii) representing additional useful information beyond just the contact location as was the case in VIRDO.
Revised: VIRDO [1] in terms of both (i) no longer needing known contact patch locations, unlocking the capability to run VIRDO in the real-world, and (ii) representing additional useful information beyond just the contact location as was the case in VIRDO.

Original: In more detail, VIRDO [1] used ground truth contact patches encoded by a PointNet encoder [40] directly which required access to this privileged information.
Revised: In more detail, VIRDO [1] uses ground truth contact patches encoded by a PointNet encoder [39] directly which required access to this privileged information.

Original: Here, the contact patch and encoder are replaced by the learnable contact latent vector.
Revised: Here, the contact patch and encoder are replaced by the learnable contact latent vector.

Original: The Action Module ( A ) predicts the next step’s boundary conditions given the current (latent) state and robot action:
Revised: The Action Module ( A ) predicts the next step’s boundary conditions given the current (latent) state and robot action:

Original: A ( α , z t , a t )
Revised: A ( α , z t , a t )

Original: =
Revised: =

Original: ˆ f t +1 , ˆ c t +1 where ˆ f t +1 ∈ R 6 is the predicted reaction wrench at the wrist,
Revised: ˆ f t +1 , ˆ c t +1 where ˆ f t +1 ∈ R 6 is the predicted reaction wrench at the wrist,

Original: ˆ c t
Revised: ˆ c t

Original: +

Revised: +

Original: is the predicted contact latent vector, and a t ∈ R 6 is the robot action (Cartesian displacement).
Revised: is the predicted contact latent vector, and a t ∈ R 6 is the robot action (Cartesian displacement).

Original: This predictive capability enables state-estimation and downstream tasks such as contact location detection crucial for deformable object manipulation.
Revised: This predictive capability enables state-estimation and downstream tasks such as contact location detection crucial for deformable object manipulation.

Original: Though we did not include in this paper, the predictive ability of VIRDO++ also enables planning for deformable object manipulation.
Revised: Though we did not include in this paper, the predictive ability of VIRDO++ also enables planning for deformable object manipulation.

Original: VIRDO lacked this functionality and could only infer the object geometry statically.
Revised: VIRDO lacked this functionality and could only infer the object geometry statically.

Original: 3.2 Geometry Representation
Revised: 3.2 Geometry Representation

Original: VIRDO++ decouples the 3D geometric representation of deformable objects into an undeformed (nominal) shape signed-distance field (SDF) and a set of deformation fields, similar to [1, 39].
Revised: VIRDO++ decouples the 3D geometric representation of deformable objects into an undeformed (nominal) shape signed-distance field (SDF) and a set of deformation fields, similar to [1, 38].

Original: The deformation field is a 3D vector field that, when summed with the deformed object SDF, results in the nominal shape SDF.
Revised: The deformation field is a 3D vector field that, when summed with the deformed object SDF, results in the nominal shape SDF.

Original: More precisely, any query point x ∈ R 3 in the robot wrist frame belonging to the deformed shape SDF can be mapped back to the corresponding point x ′ of the nominal shape
Revised: More precisely, any query point x ∈ R 3 in the robot wrist frame belonging to the deformed shape SDF can be mapped back to the corresponding point x ′ of the nominal shape

Original: SDF by applying the point-wise deformation field ∆ x as x + ∆ x = x ′ .
Revised: SDF by applying the point-wise deformation field ∆ x as x + ∆ x = x ′ .

Original: In more detail:

Original: Nominal Shape Representation: The Object Module O ( x ′ ) is the parametric representation of the nominal shape SDF – the object shape with no external contacts corresponding to ∆ x = 0 .
Revised: Nominal Shape Representation: The Object Module O ( x ′ ) is the parametric representation of the nominal shape SDF – the object shape with no external contacts corresponding to ∆ x = 0 .

Original: Here, we use a feedforward neural network to parameterize the SDF [38] and write O ( x ′ | Ψ
Revised: Here, we use a feedforward neural network to parameterize the SDF [37] and write O ( x ′ | Ψ

Original: o ( α ))
Revised: o ( α ))

Original: =
Revised: =

Original: s where s is the signed-distance at query point x ′ and Ψ
Revised: s where s is the signed-distance at query point x ′ and Ψ

Original: o
Revised: o

Original: ( α ) is a hyper-network that predicts the weights of O conditioned on the object code α .
Revised: ( α ) is a hyper-network that predicts the weights of O conditioned on the object code α .

Original: The object code and hyper-network weights are learned end-to-end in an auto-decoder approach [38].
Revised: The object code and hyper-network weights are learned end-to-end in an auto-decoder approach [37].

Original: We simplify our notation to O ( x ′ ) for the remainder.
Revised: We simplify our notation to O ( x ′ ) for the remainder.

Original: Deformation Field Representation: The generalized geometric representation of a deformable object is given by SDF ( x ) =
Revised: Deformation Field Representation: The generalized geometric representation of a deformable object is given by SDF ( x ) =

Original: O Φ o (cid:0) x
Revised: O Φ o (cid:0) x

Original: + D ( x | Ψ d ( z t , α t )) (cid:1)
Revised: + D ( x | Ψ d ( z t , α t )) (cid:1)

Original: =
Revised: =

Original: s .
Revised: s .

Original: Here, the Deformation Module
Revised: Here, the Deformation Module

Original: D ( x | Ψ d ( z t , α t )) =
Revised: D ( x | Ψ d ( z t , α t )) =

Original: ∆ x produces an object’s deformation field given force and object code pairs.
Revised: ∆ x produces an object’s deformation field given force and object code pairs.

Original: Similar to the Object Module, Deformation Module’s weights are given by the hyper-network Ψ d .
Revised: Similar to the Object Module, Deformation Module’s weights are given by the hyper-network Ψ d .

Original: 3.3 Training and Loss Formulation
Revised: 3.3 Training and Loss Formulation

Original: The training dataset for VIRDO++ is a set of trajectories S
Revised: The training dataset for VIRDO++ is a set of trajectories S

Original: = {T 1 , T 2 , ..., T N } , where each trajectory T j
Revised: = {T 1 , T 2 , ..., T N } , where each trajectory T j

Original: =
Revised: =

Original: ( p 0 , f 0 , P 0 , a 0 ) , ( p 1 , f 1 , P 1 , a 1 ) , ..., ( p k , f k , P k , a k ) } has k sequential observation and action tuples.
Revised: ( p 0 , f 0 , P 0 , a 0 ) , ( p 1 , f 1 , P 1 , a 1 ) , ..., ( p k , f k , P k , a k ) } has k sequential observation and action tuples.

Original: Here P j is a partial point cloud.
Revised: Here P j is a partial point cloud.

Original: Our first step is to pretrain a nominal shape representation (i.e., hyper-network Ψ o and object code α ).
Revised: Our first step is to pretrain a nominal shape representation (i.e., hyper-network Ψ o and object code α ).

Original: The loss for training m nominal shapes is L nominal =
Revised: The loss for training m nominal shapes is L nominal =

Original: L sdf + λ 1 L latent + λ 2 L hyper as in [1], where L latent = (cid:80) mi =1 ∥ α i ∥ 2 ,
Revised: L sdf + λ 1 L latent + λ 2 L hyper as in [1], where L latent = (cid:80) mi =1 ∥ α i ∥ 2 ,

Original: L hyper = (cid:80) mi =1 ∥
Revised: L hyper = (cid:80) mi =1 ∥

Original: Ψ o ( α i ) ∥ 2 , and:
Revised: Ψ o ( α i ) ∥ 2 , and:

Original: L sdf = m (cid:88) i = (cid:16) (cid:88) ¯ x ∈ Ω | clamp ( O i ( x ) , δ ) − clamp ( s ∗ , δ )
Revised: L sdf = m (cid:88) i = (cid:16) (cid:88) ¯ x ∈ Ω | clamp ( O i ( x ) , δ ) − clamp ( s ∗ , δ )

Original: | + λ (cid:88) ¯ x ∈
Revised: | + λ (cid:88) ¯ x ∈

Original: Ω(1 − ⟨∇
Revised: Ω(1 − ⟨∇

Original: O i ( x ) , n
Revised: O i ( x ) , n

Original: [Equation]
Revised: [Equation]

Original: n ∗ is the ground truth normal vector, δ is a parameter that clips SDF predictions, and s ∗ is the ground truth signed distance.
Revised: n ∗ is the ground truth normal vector, δ is a parameter that clips SDF predictions, and s ∗ is the ground truth signed distance.

Original: After the pretraining, we train the rest of the modules using the three losses L geo , L pred and L reg summed over a fixed horizon ( w ) while freezing Ψ o and the object code α :
Revised: After the pretraining, we train the rest of the modules using the three losses L geo , L pred and L reg summed over a fixed horizon ( w ) while freezing Ψ o and the object code α :

Original: [Equation]
Revised: [Equation]

Original: The geometry representation loss is defined as:
Revised: The geometry representation loss is defined as:

Original: [Equation]
Revised: [Equation]

Original: where x ′ = x + D Ψ d ( x ) , Ω is the 3D querying space, Ω 0 ⊂ Ω is the on-surface region, P :=
Revised: where x ′ = x + D Ψ d ( x ) , Ω is the 3D querying space, Ω 0 ⊂ Ω is the on-surface region, P :=

Original: { p | p ∈ Ω 0 } is an unordered set of the on-surface points, and ¯ P ∗ is the ground truth nominal shape point cloud.
Revised: { p | p ∈ Ω 0 } is an unordered set of the on-surface points, and ¯ P ∗ is the ground truth nominal shape point cloud.

Original: Next, we add a reaction wrench prediction loss as L pred =
Revised: Next, we add a reaction wrench prediction loss as L pred =

Original: ∥
Revised: ∥

Original: f t +1 − ˆ f t +1 ∥ .
Revised: f t +1 − ˆ f t +1 ∥ .

Original: The prediction loss for ˆ c t +1 is implicitly handled by Eq.
Revised: The prediction loss for ˆ c t +1 is implicitly handled by Eq.

Original: 2 because the contact latent vector prediction is recursively used as next step’s input during training.
Revised: 2 because the contact latent vector prediction is recursively used as next step’s input during training.

Original: In contrast, we do replace ˆ f t +1 with real measurement in Eq. 2.
Revised: In contrast, we do replace ˆ f t +1 with real measurement in Eq. 2.

Original: Finally, regularization losses L regt =
Revised: Finally, regularization losses L regt =

Original: λ 9 ∥
Revised: λ 9 ∥

Original: z t ∥ 2 + λ 10 ∥ c t ∥ 

Revised: z t ∥ 2 + λ 10 ∥ c t ∥

Original: + λ 11 ∥
Revised: + λ 11 ∥

Original: Ψ d ( α , z t ) ∥ 2 , which regularize the force codes, contact latent vector, and the weights of D respectively.
Revised: Ψ d ( α , z t ) ∥ 2 , which regularize the force codes, contact latent vector, and the weights of D respectively.

Original: 3.4 Inference and State-estimation Algorithm
Revised: 3.4 Inference and State-estimation Algorithm

Original: To maintain a belief distribution over the object deformation, we develop a particle filter-based inference algorithm, Alg. 1.
Revised: To maintain a belief distribution over the object deformation, we develop a particle filter-based inference algorithm, Alg. 1.

Original: Each particle represents a contact latent vector which can be used to reconstruct the full 3D object geometry.
Revised: Each particle represents a contact latent vector which can be used to reconstruct the full 3D object geometry.

Original: The Refine step updates the particle set by performing gradient descent on min ˆ C t L geo ( P t , ˆ C t , f t , p t ) with the visible object point cloud.
Revised: The Refine step updates the particle set by performing gradient descent on min ˆ C t L geo ( P t , ˆ C t , f t , p t ) with the visible object point cloud.

Original: The particles are then weighted according to their wrench error prediction with Weight-Function defined as w it =
Revised: The particles are then weighted according to their wrench error prediction with Weight-Function defined as w it =

Original: exp ( − γ ( ˆ f it − f t )) and resampled with probability proportional to this weight vector.
Revised: exp ( − γ ( ˆ f it − f t )) and resampled with probability proportional to this weight vector.

Original: Here, ˆ f it is the reaction wrench prediction of the i th particle and f t is a ground truth measurement.
Revised: Here, ˆ f it is the reaction wrench prediction of the i th particle and f t is a ground truth measurement.

Original: Finally, the particles are propagated through the Force and Action Modules, a small amount of uncertainties is applied ∼ N (0 , 0 . 01) to the particles and the algorithm is repeated.
Revised: Finally, the particles are propagated through the Force and Action Modules, a small amount of uncertainties is applied ∼ N (0 , 0 . 01) to the particles and the algorithm is repeated.

Original: (see algorithm variations in
Revised: (see algorithm variations in

Original: Appendix Sec.
Revised: Appendix Sec.

Original: A.2.2)
Revised: A.2.2)

Original: 1: procedure 2: ˆC 0 ← { ˆ c 00 , ˆ c 10 , ..., ˆ c n − 1} ∼ N (0 , 0 . 01)
Revised: 1: procedure 2: ˆC 0 ← { ˆ c 00 , ˆ c 10 , ..., ˆ c n − 1} ∼ N (0 , 0 . 01)

Original: ▷ Init Particles 3: for each scan in Trajectory Length do 4:
Revised: ▷ Init Particles 3: for each scan in Trajectory Length do 4:

Original: C t ← Refine ( P t , ˆC t , f t , p t ) 5: w t ← Weight-Function ( f t , ˆ f 0 t , ˆ f 1 t , ..., ˆ f n − 1 t ) 6: ¯C t ← Re-sampling ( w t , C t ) 7:
Revised: C t ← Refine ( P t , ˆC t , f t , p t ) 5: w t ← Weight-Function ( f t , ˆ f 0 t , ˆ f 1 t , ..., ˆ f n − 1 t ) 6: ¯C t ← Re-sampling ( w t , C t ) 7:

Original: Z t ← Force-Module ( ¯C t , f t , p t , α ) 8: { ( ˆ f 0 t +1 , ˆ c 0 t +1 ) , ( ˆ f 1 t +1 , ˆ c 1 t +1 ) , ..., ( ˆf n − 1 t
Revised: Z t ← Force-Module ( ¯C t , f t , p t , α ) 8: { ( ˆ f 0 t +1 , ˆ c 0 t +1 ) , ( ˆ f 1 t +1 , ˆ c 1 t +1 ) , ..., ( ˆf n − 1 t

Original: +1 , ˆ c n − 1 t +1 ) } ← Action-Module ( Z t , α , a t ) 9: t ← t +10: end for 11: end procedure
Revised: +1 , ˆ c n − 1 t +1 ) } ← Action-Module ( Z t , α , a t ) 9: t ← t +10: end for 11: end procedure

Original:  Experiments
Revised:  Experiments

Original: For real robot experiments, we investigate two classes of objects: spatulas and bike chains .
Revised: For real robot experiments, we investigate two classes of objects: spatulas and bike chains .

Original: These objects exhibit different deformation behaviors: (i) spatulas are representative of bendable elastic objects that retain their shape when external forces are removed, whereas (ii) bike chains (with the 2 end points controlled by a bi-manual manipulator) can undergo significantly larger deformations (e.g., from gravity) – requiring algorithms that can reason over free-space while estimating incontact shapes and forces.
Revised: These objects exhibit different deformation behaviors: (i) spatulas are representative of bendable elastic objects that retain their shape when external forces are removed, whereas (ii) bike chains (with the 2 end points controlled by a bi-manual manipulator) can undergo significantly larger deformations (e.g., from gravity) – requiring algorithms that can reason over free-space while estimating incontact shapes and forces.

Original: Spatula deformations are generally local and experience significant reaction wrenches, while chain deformations are much larger but with much smaller reaction wrenches.
Revised: Spatula deformations are generally local and experience significant reaction wrenches, while chain deformations are much larger but with much smaller reaction wrenches.

Original: 4.1 Spatula Manipulation Dataset
Revised: 4.1 Spatula Manipulation Dataset

Original: [Figure]
Revised: [Figure]

Original: Train and test objects are shown in Fig.
Revised: Train and test objects are shown in Fig.

Original: The training set consists ofspatulas, each with 672 samples ( N = 28 , k = 24 ), using 9:1 train/test split (25 train and 3 test trajectories) for representation learning.
Revised: The training set consists ofspatulas, each with 672 samples ( N = 28 , k = 24 ), using 9:1 train/test split (25 train and 3 test trajectories) for representation learning.

Original: The test dataset consists of 2 held-out spatulas, each with 72 samples ( N = 3 , k = 24 ), and is used to evaluate zero-shot transfer and not used for representation learning.
Revised: The test dataset consists of 2 held-out spatulas, each with 72 samples ( N = 3 , k = 24 ), and is used to evaluate zero-shot transfer and not used for representation learning.

Original: At the beginning of each trajectory ( k = 0 ), the end-effector moves to a random world Cartesian position in a bounding box of size (0 . 1 × 0 .
Revised: At the beginning of each trajectory ( t = 0 ), the end-effector moves to a random world Cartesian position in a bounding box of size (0 . 1 × 0 .

Original: 24 × 0 .
Revised: 24 × 0 .

Original: 06) meters and the tool is brought into contact with the table.
Revised: 06) meters and the tool is brought into contact with the table.

Original: Within a trajectory, the tool maintains contact while sampling a random action from ( x, y, z, r, p, y )
Revised: Within a trajectory, the tool maintains contact while sampling a random action from ( x, y, z, r, p, y )

Original: =
Revised: =

Original: ± (0 . 02 , 0 . 02 , 0 . 01 , 0 . 05 , 0 . 04) where translations are in meters and rotations in radians.
Revised: ± (0 . 02 , 0 . 02 , 0 . 01 , 0 . 05 , 0 . 04) where translations are in meters and rotations in radians.

Original: Wrench measurements are averaged over 0.05 [s] and the point cloud is registered to the wrist frame.
Revised: Wrench measurements are averaged over 0.05 [s] and the point cloud is registered to the wrist frame.

Original: The method only requires a full point cloud from the nominal geometry of the object.
Revised: The method only requires a full point cloud from the nominal geometry of the object.

Original: This is collected by holding the object in front of the camera and rotating.
Revised: This is collected by holding the object in front of the camera and rotating.

Original: Deformed object geometries are collected from a single camera viewing the front side of the spatulas without occlusions and are all partial.
Revised: Deformed object geometries are collected from a single camera viewing the front side of the spatulas without occlusions and are all partial.

Original: P is then the result of segmenting the object pointcloud and normalizing the scale into a bounding box of size 2 × 2 × 2[ m ] centered around (0 , 0 , 0) .
Revised: P is then the result of segmenting the object pointcloud and normalizing the scale into a bounding box of size 2 × 2 × 2[ m ] centered around (0 , 0 , 0) .

Original: The test dataset point clouds P contain occlusions at the bottom 0 .
Revised: The test dataset point clouds P contain occlusions at the bottom 0 .

Original: 15 × spatula-height as shown in the top observation panel in Fig.
Revised: 15 × spatula-height as shown in the top observation panel in Fig.

Original: For hardware, we use Franka Emika’s Panda, wrist mounted ATI Gamma F/T sensor, and Photoneo PhoXi 3D L.
Revised: For hardware, we use Franka Emika’s Panda, wrist mounted ATI Gamma F/T sensor, and Photoneo PhoXi 3D L.

Original: 4.2 State Estimation and Predictions
Revised: 4.2 State Estimation and Predictions

Original: [Table]
Revised: [Table]

Original: Here, we evaluate the deformation prediction (Sec. 3.1) and state-estimation (Alg. 1) accuracy of VIRDO++ on training and test trajectories.
Revised: Here, we evaluate the deformation prediction (Sec. 3.1) and state-estimation (Alg. 1) accuracy of VIRDO++ on training and test trajectories.

Original: Fig.
Revised: Fig.

Original: 4 shows an example of stateestimation for a test trajectory where the ground truth (black) and reconstruction (magenta) are overlaid.
Revised: 4 shows an example of stateestimation for a test trajectory where the ground truth (black) and reconstruction (magenta) are overlaid.

Original: We note the high state-estimation accuracy indicated by the overlay agreement, where the average distance between the measured points and reconstructions is within
Revised: We note the high state-estimation accuracy indicated by the overlay agreement, where the average distance between the measured points and reconstructions is within

Original: 0.03 [ m ] of normalized scale, just 1% of the object length.
Revised: 0.03 [ m ] of normalized scale, just 1% of the object length.

Original: The quantitative results, Tab. 1, show
Revised: The quantitative results, Tab. 1, show

Original: VIRDO++ is able to faithfully predict and estimate object deformations and reaction forces despite significant occlusions.
Revised: VIRDO++ is able to faithfully predict and estimate object deformations and reaction forces despite significant occlusions.

Original: Force and torque norm errors are less than 0.567 [N], 0.

Revised: Force and torque norm errors are less than 0.567 [N], 0.

Original: [Nm] (train) and 1.130 [N], 0.

Revised: [Nm] (train) and 1.130 [N], 0.

Original: [Nm] (test) respectively (see Appendix Sec.
Revised: [Nm] (test) respectively (see Appendix Sec.

Original: A.3.3 and Sec.
Revised: A.3.3 and Sec.

Original: A.3.4 for details).
Revised: A.3.4 for details).

Original: Our ablation study, Tab. 3, helps further anchor the scale of these state-estimation and dynamics prediction errors.
Revised: Our ablation study, Tab. 3, helps further anchor the scale of these state-estimation and dynamics prediction errors.

Original: On state estimation, for example, our full VIRDO++ method achieves a Chamfer distance ( × 10 3 ) of normalized geometry is 0.93 [ m 2 ] .
Revised: On state estimation, for example, our full VIRDO++ method achieves a Chamfer distance ( × 10 3 ) of normalized geometry as 0.93 [ m 2 ] .

Original: The baseline model without a novel contact latent vector input has 27% increased chamfer distance error as 1.18 [ m 2 ] for geometry estimation.
Revised: The baseline model without a novel contact latent vector input has 27% increased chamfer distance error as 1.18 [ m 2 ] for geometry estimation.

Original: Without c t , the baseline does not reason about contact during inference, resulting in larger errors during state-estimation Further if we assume the object is rigid and reduce our state estimation to a “rigid pose estimation” pipeline, the Chamfer distance ( × 10 3 ) increases considerably to 16.19 [ m 2 ] .
Revised: Without c t , the baseline does not reason about contact during inference, resulting in larger errors during state-estimation. Further if we assume the object is rigid and reduce our state estimation to a “rigid pose estimation” pipeline, the Chamfer distance ( × 10 3 ) increases considerably to 16.19 [ m 2 ] .

Original: The geometry predictions are slightly less accurate than state-estimation because the geometry prediction are computed by the learned dynamics using previous observation and action, while stateestimation uses the current measurement to update the prediction.
Revised: The geometry predictions are slightly less accurate than state-estimation because the geometry predictions are computed by the learned dynamics using previous observation and action, while stateestimations use the current measurement to update the prediction.

Original: Despite this, the prediction Chamfer distances are within 14% of the state estimation results.
Revised: Despite this, the prediction Chamfer distances are within 14% of the state estimation results.

Original: Moreover, the train and test trajectories results show 25% difference in Chamfer distance.
Revised: Moreover, the train and test trajectories results show 25% difference in Chamfer distance.

Original: This is interesting because the test trajectories are not only from unseen tool configurations but also with occlusions (see Dataset ).
Revised: This is interesting because the test trajectories are not only from unseen tool configurations but also with occlusions (see Dataset ).

Original: This suggests that learned dynamics of VIRDO++ generalize effectively to unseen contact formations with occlusions.
Revised: This suggests that learned dynamics of VIRDO++ generalize effectively to unseen contact formations with occlusions.

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: 4.3 Evaluation on a Downstream Manipulation Task - Extrinsic Contact Detection
Revised: 4.3 Evaluation on a Downstream Manipulation Task - Extrinsic Contact Detection

Original: Central to compliant tool manipulation is estimating and controlling the contact between the tool and environment.
Revised: Central to compliant tool manipulation is estimating and controlling the contact between the tool and environment.

Original: For example, when scraping a wok with a spatula, the robot must reason over the contact formation (e.g., point, line, or patch) to ensure the spatula is scraping properly at the desired region on the wok.
Revised: For example, when scraping a wok with a spatula, the robot must reason over the contact formation (e.g., point, line, or patch) to ensure the spatula is scraping properly at the desired region on the wok.

Original: In this section, we evaluate VIRDO++ ability to estimate extrinsic contact features; specifically, the contact line created as the spatula is brought into contact with the table.
Revised: In this section, we evaluate VIRDO++ ability to estimate extrinsic contact features; specifically, the contact line created as the spatula is brought into contact with the table.

Original: To this end, we compute the signed distance values of the tables surface ( z = 0) to compute a contact cluster.
Revised: To this end, we compute the signed distance values of the tables surface ( z = 0) to compute a contact cluster.

Original: This requires only a single feed-forward path where the input is an N × 3 matrix, where N is the number of query points.
Revised: This requires only a single feed-forward path where the input is an N × 3 matrix, where N is the number of query points.

Original: From the contact cluster, we extract a contact line by selecting the two points furthest apart similar to [41, 42, 43].
Revised: From the contact cluster, we extract a contact line by selecting the two points furthest apart similar to [40, 41, 42].

Original: The evaluation metrics are the average Euclidean distance 12 Σ 2 i =

Revised: The evaluation metrics are the average Euclidean distance 12 Σ 1 i =

Original: || l i − l ∗
Revised: || l i − l ∗

Original: i || , where l i is the detected contact lines, l ∗ i is the ground truth contact lines, and i = 0 , 1 refers to right and left ends of each contact line respectively.
Revised: i || , where l i is the detected contact lines, l ∗ i is the ground truth contact lines, and i = 0 , 1 refers to right and left ends of each contact line respectively.

Original: Ground truth contact lines were obtained from ground truth pointcloud without occlusions by applying a contact mask z ≤ ϵ .
Revised: Ground truth contact lines were obtained from ground truth pointcloud without occlusions by applying a contact mask z ≤ ϵ .

Original: This contact feature estimation task is particularly interesting as it is the inverse problem of [1]; estimating [1]’s privileged input (=contact location) by inferring c t .
Revised: This contact feature estimation task is particularly interesting as it is the inverse problem of [1]; estimating [1]’s privileged input (=contact location) by inferring c t .

Original: As shown in Tab. 2, VIRDO++ can accurately detect contact locations with less than 8 mm error on a flat table for the current and the following time step regardless of occlusions at the tip.
Revised: As shown in Tab. 2, VIRDO++ can accurately detect contact locations with less than 8 mm error on a flat table for the current and the following time step regardless of occlusions at the tip.

Original: As in
Revised: As in

Original: Sec.4.2, test trajectories from 4 spatulas with 15% occlusion were used for the evaluation.
Revised: Sec.4.2, test trajectories from 4 spatulas with 15% occlusion were used for the evaluation.

Original: Tab. shows that the average contact line detection errors from all 4 spatulas are less than 7 [mm] during estimation and are less than 8 [mm] for predictions in Euclidean error.
Revised: Tab. shows that the average contact line detection errors from all 4 spatulas are less than 7 [mm] during estimation and are less than 8 [mm] for predictions in Euclidean error.

Original: Similar to the result in
Revised: Similar to the result in

Original: [Table]
Revised: L

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]

Original: [Table]
Revised: [Table]

Original: Sec. 4.2, estimations are 

Revised: Sec. 4.2, estimations are

Original: [mm] more precise than predictions because they are optimized using the sensor measurements.
Revised: [mm] more precise than predictions because they are optimized using the sensor measurements.

Original: Considering that the total length of the spatulas are about 200 [mm], this is less than 4% of error in total length.
Revised: Considering that the total length of the spatulas are about 200 [mm], this is less than 4% of error in total length.

Original: Estimating extrinsic contacts is a challenging task for rigid object [44] and is further complicated by object compliance; however, our approach proves effective due to its high-quality 3D reconstructions (details of error distribution in Appendix Fig. 12).
Revised: Estimating extrinsic contacts is a challenging task for rigid object [43] and is further complicated by object compliance; however, our approach proves effective due to its high-quality 3D reconstructions (details of error distribution in Appendix Fig. 12).

Original: Tab.also quantifies performance against ablations (without c t and Rigid-body assumption) illustrating the importance of the contact embedding and reasoning over compliance.
Revised: Tab.also quantifies performance against ablations (without c t and Rigid-body assumption) illustrating the importance of the contact embedding and reasoning over compliance.

Original: 4.4 Generalizations Tasks
Revised: 4.4 Generalizations Tasks

Original: This section demonstrates VIRDO++  generalization ability to unseen objects and their dynamics.
Revised: This section demonstrates VIRDO++ ’s generalization ability to unseen objects and their dynamics.

Original: For each unseen object, the generalization dataset contains 1 nominal point cloud and 3 trajectories, each with 24 transitions with occlusion.
Revised: For each unseen object, the generalization dataset contains 1 nominal point cloud and 3 trajectories, each with 24 transitions with occlusion.

Original: Here, we directly apply the same pre-trained model from
Revised: Here, we directly apply the same pre-trained model from

Original: Sec.
Revised: Sec.

Original: 4.1 to the generalization test dataset.
Revised: 4.1 to the generalization test dataset.

Original: 4.4.1 Object code inference for Unseen Objects
Revised: 1 Object code inference for Unseen Objects

Original: Train Objects
Revised: Train Objects

Original: Test Objects
Revised: Test Objects

Original: VIRDO++ is able to represent nominal geometries unseen during training within distribution.
Revised: VIRDO++ is able to represent nominal geometries unseen during training within distribution.

Original: Fig.
Revised: Fig.

Original: 6 shows reconstructions of both seen and unseen object in Fig.
Revised: 6 shows reconstructions of both seen and unseen object in Fig.

Original: 3 using a single pretrained
Revised: 3 using a single pretrained

Original: Object Module.
Revised: Object Module.

Original: For the unseen objects, we start from a randomly initialized object code α ∼ N (0 , 0 . 01) and update the code by gradient descent with Adam.
Revised: For the unseen objects, we start from a randomly initialized object code α ∼ N (0 , 0 . 01) and update the code by gradient descent with Adam.

Original: We use the same loss function from
Revised: We use the same loss function from

Original: Eq. 1 with pre-trained VIRDO++ while freezing the weights of the network.
Revised: Eq. 1 with pre-trained VIRDO++ while freezing the weights of the network.

Original: The first row of Tab. 3 shows the Chamfer distance error of the unseen objects alongside training objects, indicating the relatively small drop in performance.
Revised: The first row of Tab. 3 shows the Chamfer distance error of the unseen objects alongside training objects, indicating the relatively small drop in performance.

Original: When comparing Fig.
Revised: When comparing Fig.

Original: 6 and the ground truth in Fig.
Revised: 6 and the ground truth in Fig.

Original: 3, we note that the test reconstructions are close to the ground truth aside from some fine edge details.
Revised: 3, we note that the test reconstructions are close to the ground truth aside from some fine edge details.

Original: t =
Revised: t =

Original: t =t =t = t =t =
Revised: t =t =t = t =t =

Original: 4.4.2 Zero Shot Dynamics Prediction and State-estimation Generalization
Revised: 4.4.2 Zero Shot Dynamics Prediction and State-estimation Generalization

Original: With the object codes obtained in Sec. 4.4.
Revised: With the object codes obtained in Sec. 4.4.

Original: [Table]
Revised: [Table]

Original: VIRDO++ is able to predict the unseen object’s dynamics based on their nominal shape.
Revised: VIRDO++ is able to predict the unseen object’s dynamics based on their nominal shape.

Original: Here
Revised: Here

Original: [Table]
Revised: [Table]

Original: we evaluate both prediction and state-estimation performance as in Sec.
Revised: we evaluate both prediction and state-estimation performance as in Sec.

Original: We note that the observations of the unseen objects include occlusions as discussed in Dataset .
Revised: We note that the observations of the unseen objects include occlusions as discussed in Dataset .

Original: Tab.
Revised: Tab.

Original: 3 shows comparable performance between training and test objects with a small drop in accuracy, highlighting our model’s generalization capability.
Revised: 3 shows comparable performance between training and test objects with a small drop in accuracy, highlighting our model’s generalization capability.

Original: Sec.
Revised: Sec.

Original: A.3.6 in Appendix provides intuition for the Chamfer distance values.
Revised: A.3.6 in Appendix provides intuition for the Chamfer distance values.

Original: Further, we evaluate VIRDO++ ’s ability to perform high-fidelity geometry prediction and estimation even in novel environment (Wok scraping) without 3D model.
Revised: Further, we evaluate VIRDO++ ’s ability to perform high-fidelity geometry prediction and estimation even in novel environment (Wok scraping) without 3D model.

Original: Details can be found in Appendix Sec.
Revised: Details can be found in Appendix Sec.

Original: A.
Revised: A.

Original: [Table] 
Revised: [Table] [Table]

Original: 4.5 Bike Chain Application
Revised: 4.5 Bike Chain Application

Original: In this experiment the chain ends are grasped by two arms.
Revised: In this experiment the chain ends are grasped by two arms.

Original: One end of the chain is moved randomly through the vertices of an equally-spaced 6 × 6 × 6 grid fitting inside of a 0 .
Revised: One end of the chain is moved randomly through the vertices of an equally-spaced 6 × 6 × 6 grid fitting inside of a 0 .

Original: 1 × 0 .
Revised: 1 × 0 .

Original: 18 × 0 .meter bounding box.
Revised: 18 × 0 .meter bounding box.

Original: The visited vertices of this grid make up the training dataset.
Revised: The visited vertices of this grid make up the training dataset.

Original: The test dataset is generated by sampling 20 random positions uniformly from the bounding box.
Revised: The test dataset is generated by sampling 20 random positions uniformly from the bounding box.

Original: The end-effector orientations were fixed; therefore, p t , a t ∈ R 3 .
Revised: The end-effector orientations were fixed; therefore, p t , a t ∈ R 3 .

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 7, VIRDO++ is able to predict/estimate both free-space and in-contact chain geometry.
Revised: 7, VIRDO++ is able to predict/estimate both free-space and in-contact chain geometry.

Original: The in-contact phase occurs when the arm moves sufficiently far down such that the chain lays on the table.
Revised: The in-contact phase occurs when the arm moves sufficiently far down such that the chain lays on the table.

Original: We report the quantitative results of geometry estimation and dynamics prediction in Tab. 1 and Appendix Sec.
Revised: We report the quantitative results of geometry estimation and dynamics prediction in Tab. 1 and Appendix Sec.

Original: A.3.3.
Revised: A.3.3.

Original: The empirical results are comparable to spatula experiments despite significantly larger shape variations.
Revised: The empirical results are comparable to spatula experiments despite significantly larger shape variations.

Original: Details on wrench prediction errors in Appendix Sec.
Revised: Details on wrench prediction errors in Appendix Sec.

Original: A.3.4.
Revised: A.3.4.

Original:  Limitations
Revised:  Limitations

Original: Our proposed approach has a number of limitations: 1) Non-rigid grasps: VIRDO++ assumes that the gripper is rigidly holding the object.
Revised: Our proposed approach has a number of limitations: 1) Non-rigid grasps: VIRDO++ assumes that the gripper is rigidly holding the object.

Original: When it comes to non-rigid grasping, we need to estimate relative motion of the objects w.r.t the gripper and the force transmission is now a function of this relative motion.
Revised: When it comes to non-rigid grasping, we need to estimate relative motion of the objects w.r.t the gripper and the force transmission is now a function of this relative motion.

Original: One potential solution is to use collocated tactile sensor feedback [45, 46, 47] with F/T sensing to reason over reaction forces conditioned on the grasp.
Revised: One potential solution is to use collocated tactile sensor feedback [44, 45, 46] with F/T sensing to reason over reaction forces conditioned on the grasp.

Original: 2) Inertial effects: our approach assumes quasi-static manipulation (negligible accelerations).
Revised: 2) Inertial effects: our approach assumes quasi-static manipulation (negligible accelerations).

Original: During fast motions, the tools’ inertial effects may also impact deformations and wrench measurements.
Revised: During fast motions, the tools’ inertial effects may also impact deformations and wrench measurements.

Original: When applicable, we suggest supplementing VIRDO++ ’s input with end-effector accelerations.
Revised: When applicable, we suggest supplementing VIRDO++ ’s input with end-effector accelerations.

Original: 3) Generalization to novel environments: While we provide an example of environment generalization (wok experiments Fig. 14), the performance is lower than our other demonstrated applications.
Revised: 3) Generalization to novel environments: While we provide an example of environment generalization ( Appendix Sec. A.4), the performance is lower than our other demonstrated applications.

Original: VIRDO++ may benefit from a 3D model of the environment to integrate inter-penetration losses [24] or collision checkers [48].
Revised: VIRDO++ may benefit from a 3D model of the environment to integrate inter-penetration losses [23] or collision checkers [47].

Original: Acknowledgments
Revised: Acknowledgments

Original: This work was in part supported by Robotics at Google under the Google Faculty Research Award 2021 and the National Science Foundation (NSF) grant NRI-2220876.
Revised: This work was in part supported by Robotics at Google under the Google Faculty Research Award 2021 and the National Science Foundation (NSF) grant NRI-2220876.

Original: Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation 
Revised: Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation A.1 Training Details Tab. 4 contains training parameters.

Revised: In addition, we used 2,000 on-surface points and 20,000 off-surfaces for training VIRDO++ .

Revised: We used the Adam optimizer with learning rate = 10 − 4 and epoch = 200.

Revised: l c λ λλλλλλλλλλλ 5e 1e 1e 1e  1e 1e 5e 3e 1e 1e 1e

Revised: A.2 Inference Details A.2.1 Parameters & Hardware

Revised: For the inference, we use number of particles n = 40 , resampling parameter γ = 0 . 7 , and epoch e = 10 .

Original: {+This results in inference time = 0.
+}
Revised: {+This results in inference time = 0.
+}

Revised: [s] with GPU = NVIDIA A6000 and CPU= 32-Core 3.50 GHz AMD.

Revised: We note that one can balance the state-estimation accuracy between the inference time by changing the number of particles and number of epochs.

Revised: Here, we note that the epochs need not to be large due to Alg. 1 which provides a close-to-ground-truth initial guess on the object’s state.

Revised: A.2.2 Inference Algorithm Variations

Revised: Algorithm

Revised: 1: function R E - SAMPLING ( w t , C t ) 2:

Revised: Draw { ¯ C 0 t , ¯ C 1 t , ..., ¯ C k − 1 t } ∼ N (0 , 0 . 01)

Revised: ▷ Exploration 3:

Revised: Draw { ¯ C k t , ¯ C k +1 t , ..., ¯ C N − 1 t } ∼ { ( C 0 t , w 0 t ) , ( C 1 t , w 1 t ) ..., ( C N − 1 t , w N − 1 t ) } ▷ Exploitation 4: return C t 5: end function

Revised: The input of the resampling function in Alg.

Revised: 1 is the beliefs over the current states ( C t ) and the corresponding reliability, a.k.a the sampling weight ( w t ).

Revised: The resampling function refines C t to ¯C t through Alg. 2.

Revised: In the

Revised: [Table]

Revised: Our ablation study (presented in Tab. 5) aims to find an optimal balance between exploration and exploitation.

Revised: Here, the “Force-Prediction” is calculated as ∥ ˆ f t [: 3] − f t [: 3] ∥ and “Torque-Prediction” is calculated as ∥ ˆ f t [3 :] − f t [3 :] ∥ .

Revised: We found that β

Revised: = 0 and 25% has comparable state estimation results.

Revised: Geometry prediction is the most accurate when β = 0% ; however, wrench predictions are most accurate when β

Revised: = 25% .

Revised: When compared to β = 0% , β = 25% has a 10% increased geometry prediction error and 9 − 10% reduced wrench prediction error.

Revised: A.3 Experiment Details A.3.1 Differences in Physical Properties of Spatulas

Revised: In this section, we illustrate how the train/test spatulas deform relative to each other when brought into contact with the table in a similar configurations (affected by the arm impedance controller) and same normal force (here, 6 [N] along the z axis).

Revised: Fig.

Revised: 8 shows a wide distribution of deformations achieved within the same object category.

Revised: With the same force, train object 1 barely deforms and is close to it’s nominal shape, whereas train object 3 bends significantly.

Revised: This difference in physical properties makes generalization very difficult, even within the same object class.

Revised: For example, test object 2 visually resembles train object 4; however, it has stiffness properties more similar to train object 2.

Revised: This has lead to VIRDO++ producing higher errors in the generalization task for test 2 objects.

Revised: A.3.2 Coordinate System and Action Distribution

Revised: Pointcloud are collected with respect to a reference frame defined at the end of the F/T sensor mounted at the robot’s wrist.

Revised: Actions are the Cartesian motion of the end-effector of the robot with respect to the base frame,

Revised: Fig

Revised: A.3.3 Distribution of Geometry and Wrench State Estimation and Predictions

Revised: Fig.

Revised: 11 shows detailed distribution of errors reported in Tab. 1.

Revised: The Kernel Density Estimate (KDE) plot shows that the errors are normally distributed.

Revised: The training error mean is smaller than the test error for all graphs in both demos.

Revised: In addition, geometry estimation is more accurate than geometry predictions as it uses sensor measurements.

Revised: For example, in the spatula demo, the Chamfer distance error at the peak geometry estimation from the training set (green dashed line) is smaller than the test set (solid dashed line) and is also smaller than the prediction estimates (yellow dashed line).

Revised: =

Revised: A.3.4 Reaction Wrench Prediction Errors

Revised: Tab. 6 shows the wrench predictions results of trained objects (Tab. 3’s first column).

Revised: Because we use L2 norm for reaction wrench prediction during training, error distributions are centered around 0, and the degree of errors appears as standard deviation.

Revised: The standard deviation in error for the chain task is approximately an order of magnitude smaller than the spatula.

Revised: We hypothesize that this because i) the ground truth wrench is an order of magnitude smaller than the spatula and ii) the action space dimension is half the spatula resulting in a simpler model to learn.

Revised: [Table]

Revised: A.3.5 Details o Contact Detection Errors

Revised: Fig.

Revised: 12 contains detailed error distributions of Tab. 2 from the main text.

Revised: VIRDO++ (green)’s

Revised: Chamfer dis- tance has a significantly smaller mean and standard deviation compared to the baseline models without contact embedding and compliance assumption.

Revised: One interesting observation is that the geometry prediction (yellow dashed line) is located left to the geometry estimation (solid yellow line).

Revised: This is because the model without c t produces deterministic reconstructions as a function of p t and f t , leaving no room for improving state estimation with the Bayesian filtering algorithm.

Revised: A.3.6 Details on Generalization to Novel Objects Dynamics

Revised: Fig.

Original: {+13 is a detailed Chamfer distance distribution of Tab. 
+}
Revised: {+13 is a detailed Chamfer distance distribution of Tab. 
+}

Revised: from the main text and the of each Chamfer distance loss.

Revised: The purpose of reconstructions visualized in Fig.

Revised: 13 (right) is to further provide an intuition of each error and how to interpret its magnitude.

Revised: For example, the Chamfer distance value of 0.002 for the unseen objects is about double the amount of the training objects’ average state estimation error; however, in visualization, the reconstructions can reasonably predict the ground truth’s curvature.

Revised: We observed that the majority of errors are due to small undesirable artifacts at the edges of objects which do not significantly impact the general trend of deformations.

Revised: A.4 Generalization to Novel Environments

Revised: In this section, we show VIRDO++ generalization capability to novel environments.

Revised: Here, the novel envi- ronment is a wok scraping task.

Revised: The wok presents high curvatures producing greater variances in contact formations with respect to the flat table.

Revised: Here, we add a training dataset with 2 scraping trajectories on the wok and perform state estimation and predictions given a test trajectory with 13 transitions.

Revised: As in Fig.

Revised: VIRDO++ provides high-fidelity dense geometry of deformable objects in high occlusion cases without a 3D model or configuration of the environment; however, due to the complex environment, it occasionally produces undesired artifacts (2 out of 13 trans.).

Revised: [Figure]

Revised: A.5 Ablation study on design choices A.5.1 Explicit Geometry Representation

Revised: In this section, we compare VIRDO++ with explicit models by replacing the geometry representations (defor- mation and object module) with GRNet’s pointcloud encoder-decoder structure [48].

Revised: Here, we use the same force module as VIRDO++ because the baseline does not have a multi-modal sensory encoder.

Revised: The explicit dynamics model directly takes in a partial pointcloud at time t to generate a dense pointcloud at time t +1 .

Revised: The inputs to the dynamics model are pointcloud at the current time step P t , force code z t , contact latent vector c t , and action a t and the output is full pointcloud at the next time step P t +1 .

Revised: While P t is directly input to [48]’s encoder, and other inputs are concatenated to the bottle-neck features.

Revised: Here, we used the same dataset in Sec. 4.1.

Revised: For the loss function, we replaced L geot to Chamfer distance as [48] proposed.

Revised: The second column of Tab.

Revised: 7 shows the result of the explicit dynamics model based on [48], where the learned explicit model showed better result than VIRDO++ for training trajectories.

Revised: However, the performance dropped significantly given the pointcloud with occlusions from the test trajectories.

Revised: We speculate that because the explicit dynamics model directly consumes the partial pointcloud to roll out the dynamics, it is more susceptible to visual outliers caused by occlusions.

Revised: However, VIRDO++ is more robust to occlusions due to its implicit nature, using partial pointcloud implicitely via solving the optimization problem in Sec. 3.

Revised: Besides dealing with occlusions, VIRDO++ ’s implicit dynamics model is also advantageous over the explicit counterparts in multi-object interactions.

Revised: Specifically, the detection of contact and collision at a query point can easily be done by evaluating the sign-distances from each object’s signed distance field at a time complexity of O ( number of objects ) .

Revised: On the other hand, explicit geometric representations require special contact/collision detection techniques such as k-d tree [49] or sphere coverings [50].

Revised: These methods have higher time complex- ities dependent on the length of each point cloud.

Revised: In Sec.4.3, we demonstrate the accurateness of

Revised: VIRDO++ for detecting contact location.

Revised: A.5.2 Implicit Geometry Representation with Different Model Architectures

Revised: In this section, we evaluate variations of VIRDO’s geometry representation using DeepSDF [37], SIREN [36], and a positional encoder [52].

Revised: In this ablation study we first attempted to replace the VIRDO++ ’s entire geometry representation modules (i.e., both deformation and shape module) with [37], [36], and [52], then train each model end to end with the VIRDO++ ’s force and action modules.

Revised: Under this condition, we set the input as ( x , α, z t ) and the output as a signed distance function s ∈ R , where x is the query point instead of the entire pointcloud as Sec.

Revised: A.5.1.

Revised: However, training end to end without residual layers resulted in failure of convergence for all variations.

Revised: We observed that fixing the weights of the pre-trained object modules and learning only the residuals (defor- mations) significantly helped the models converge.

Revised: We chose to replace only the deformation module with the proposed variations and compare performance.

Revised: The inputs are ( x , α, z t ) and the output is ∆ x .

Revised: We used the

Revised: [Table]

Revised: : Here, we replaced the geometry representation with SIREN’s hypernetwork and trained it with the modules.

Revised: As suggested in the paper, we implemented SIREN with 5 layers and hidden features of size 256.

Original: {+HyperNetwork (ReLU MLP) was initialized with Kaiming initialization, and the HypoNetwork was initialized as w i ∼ U ( − 
+}
Revised: {+HyperNetwork (ReLU MLP) was initialized with Kaiming initialization, and the HypoNetwork was initialized as w i ∼ U ( − 
+}

Revised: /l, +1 /l ) , where w i is the network’s weight and l is the total number of network parameters.

Revised: We found that the sine activation function generates bumpy textures resulting in high reconstruction errors for geometry estimation and predictions.

Revised: Positional Encoder: For this experiment, we add a positional encoder from [52] right before VIRDO++ ’s deformation modules.

Revised: Similar to what we observed from SIREN, adding sine or cosine non-linearity to the network can produce undesirable artifacts.

Revised: We found that the positional encoder generate floating blobs the space, resulting in much higher errors the original

Revised: ++ .