Original: Learning Models as Functionals of Signed-Distance Fields for Manipulation Planning
Revised: Learning Models as Functionals of Signed-Distance Fields for Manipulation Planning

Original: Abstract: This work proposes an optimization-based manipulation planning framework where the objectives are learned functionals of signed-distance ﬁelds that represent objects in the scene.
Revised: Abstract: This work proposes an optimization-based manipulation planning framework where the objectives are learned functionals of signed-distance ﬁelds that represent objects in the scene.

Original: Most manipulation planning approaches rely on analytical models and carefully chosen abstractions/state-spaces to be effective.
Revised: Most manipulation planning approaches rely on analytical models and carefully chosen abstractions/state-spaces to be effective.

Original: A central question is how models can be obtained from data that are not primarily accurate in their predictions, but, more importantly, enable efﬁcient reasoning within a planning framework, while at the same time being closely coupled to perception spaces.
Revised: A central question is how models can be obtained from data that are not primarily accurate in their predictions, but, more importantly, enable efﬁcient reasoning within a planning framework, while at the same time being closely coupled to perception spaces.

Original: We show that representing objects as signed-distance ﬁelds not only enables to learn and represent a variety of models with higher accuracy compared to point-cloud and occupancy measure representations, but also that
Revised: We show that representing objects as signed-distance ﬁelds not only enables to learn and represent a variety of models with higher accuracy compared to point-cloud and occupancy measure representations, but also that

Original: SDF-based models are suitable for optimization-based planning.
Revised: SDF-based models are suitable for optimization-based planning.

Original: To demonstrate the versatility of our approach, we learn both kinematic and dynamic models to solve tasks that involve hanging mugs on hooks and pushing objects on a table.
Revised: To demonstrate the versatility of our approach, we learn both kinematic and dynamic models to solve tasks that involve hanging mugs on hooks and pushing objects on a table.

Original: We can unify these quite different tasks within one framework, since SDFs are the common object representation.
Revised: We can unify these quite different tasks within one framework, since SDFs are the common object representation.

Original: Video: https://youtu.be/ga8Wlkss7co
Revised: Video: https://youtu.be/ga8Wlkss7co

Original:  Introduction
Revised:  Introduction

Original: Manipulation planning is challenging for multiple reasons.
Revised: Manipulation planning is challenging for multiple reasons.

Original: On the one hand, planning robot motions to solve a task can be formulated as a decision problem over a high-dimensional, non-convex space, including discrete and continuous aspects.
Revised: On the one hand, planning robot motions to solve a task can be formulated as a decision problem over a high-dimensional, non-convex space, including discrete and continuous aspects.

Original: Especially long-horizon tasks that consist of multiple manipulation steps have the property that motions have to be coordinated globally with the future goal.
Revised: Especially long-horizon tasks that consist of multiple manipulation steps have the property that motions have to be coordinated globally with the future goal.

Original: This coupling of potentially all variables requires joint reasoning and makes the problem particularly challenging [1].
Revised: This coupling of potentially all variables requires joint reasoning and makes the problem particularly challenging [1].

Original: On the other hand, the problem solving capabilities of a planning framework is inherently dependent on its underlying models.
Revised: On the other hand, the problem solving capabilities of a planning framework is inherently dependent on its underlying models.

Original: The ﬁeld of Task and Motion Planning (TAMP) has made signiﬁcant progress in solving challenging multi-step, long-horizon tasks [2], ranging from ones that involve mainly kinematic models [3, 4, 5, 6, 7] to dynamic tasks that require reasoning about forces, friction etc.
Revised: The ﬁeld of Task and Motion Planning (TAMP) has made signiﬁcant progress in solving challenging multi-step, long-horizon tasks [2], ranging from ones that involve mainly kinematic models [3, 4, 5, 6, 7] to dynamic tasks that require reasoning about forces, friction etc.

Original: based on more general dynamic equations [8, 9, 10, 11, 12, 13, 14].
Revised: based on more general dynamic equations [8, 9, 10, 11, 12, 13, 14].

Original: However, most TAMP approaches rely on carefully chosen abstractions and analytically deﬁned models in order to be successful and efﬁcient.
Revised: However, most TAMP approaches rely on carefully chosen abstractions and analytically deﬁned models in order to be successful and efﬁcient.

Original: In particular, TAMP often makes simplifying assumptions on the possible geometries of objects it can deal with to deﬁne manipulation constraints in the ﬁrst place.
Revised: In particular, TAMP often makes simplifying assumptions on the possible geometries of objects it can deal with to deﬁne manipulation constraints in the ﬁrst place.

Original: It is unclear how these models can be grounded from sensor information.
Revised: It is unclear how these models can be grounded from sensor information.

Original: To overcome these issues, a natural idea is to replace the analytic models in TAMP frameworks with learned ones.
Revised: To overcome these issues, a natural idea is to replace the analytic models in TAMP frameworks with learned ones.

Original: Recent advances in deep learning have enabled to learn predictive forward models even in high-dimensional observation spaces like images.
Revised: Recent advances in deep learning have enabled to learn predictive forward models even in high-dimensional observation spaces like images.

Original: The typical objective for learning a forward model is its predictive accuracy.
Revised: The typical objective for learning a forward model is its predictive accuracy.

Original: However, having an accurate model does not necessarily imply that a planning framework can utilize it efﬁciently.
Revised: However, having an accurate model does not necessarily imply that a planning framework can utilize it efﬁciently.

Original: While having an accurate forward prediction model might be sufﬁcient for short-horizon tasks, especially for long-horizon tasks, learned models can exhibit too high combinatorics for sampling or non-informative gradients for achieving future goals.
Revised: While having an accurate forward prediction model might be sufﬁcient for short-horizon tasks, especially for long-horizon tasks, learned models can exhibit too high combinatorics for sampling or non-informative gradients for achieving future goals.

Original: This paper aims to address these challenges by learning models that can be used effectively by a planning framework while at the same time using a general object representation more closely related to senors spaces.
Revised: This paper aims to address these challenges by learning models that can be used effectively by a planning framework while at the same time using a general object representation more closely related to senors spaces.

Original: To realize this, we present an optimization-based TAMP framework where the objectives are learned functionals of signed-distance ﬁelds (SDFs).
Revised: To realize this, we present an optimization-based TAMP framework where the objectives are learned functionals of signed-distance ﬁelds (SDFs).

Original: The SDFs represent each object in the scene separately, while the functionals deﬁned on top of them induce constraints on possible, physically plausible interactions between the objects within a trajectory optimization problem.
Revised: The SDFs represent each object in the scene separately, while the functionals deﬁned on top of them induce constraints on possible, physically plausible interactions between the objects within a trajectory optimization problem.

Original: The task planning aspect is realized by (discrete) decisions that determine which of those functionals are active at which phase of the planning horizon.
Revised: The task planning aspect is realized by (discrete) decisions that determine which of those functionals are active at which phase of the planning horizon.

Original: We argue that representing objects as SDFs has multiple advantages.
Revised: We argue that representing objects as SDFs has multiple advantages.

Original: First, an SDF can be seen as an intermediate representation between raw perception like point-clouds or images and full state information.
Revised: First, an SDF can be seen as an intermediate representation between raw perception like point-clouds or images and full state information.

Original: While not the focus of this work, many methods have been developed to learn and obtain SDFs from, e.g., image observations of the scene.
Revised: While not the focus of this work, many methods have been developed to learn and obtain SDFs from, e.g., image observations of the scene.

Original: Further, SDFs can represent arbitrary, nonconvex geometries, which is beneﬁcial, since manipulation problems and physical phenomena often depend on the geometry of the interacting objects.
Revised: Further, SDFs can represent arbitrary, nonconvex geometries, which is beneﬁcial, since manipulation problems and physical phenomena often depend on the geometry of the interacting objects.

Original: Finally, we show that SDFs are particularly suited for learning and representing models that can later be used within a planning framework effectively.
Revised: Finally, we show that SDFs are particularly suited for learning and representing models that can later be used within a planning framework effectively.

Original: Since our models are functionals of the SDFs, the constraints can take the information about whole objects into account to reason about their geometry and therefore especially the interaction between objects.
Revised: Since our models are functionals of the SDFs, the constraints can take the information about whole objects into account to reason about their geometry and therefore especially the interaction between objects.

Original: Compared to a representation that only describes the surface of an object like point clouds or occupancy measures, a signed-distance ﬁeld also provides information about the object at distance.
Revised: Compared to a representation that only describes the surface of an object like point clouds or occupancy measures, a signed-distance ﬁeld also provides information about the object at distance.

Original: As we experimentally show, this not only leads to models that perform better in their prediction accuracy compared to models learned on top of point-cloud or occupancy object representations, but SDFs also enable the functionals/learned models to have more useful gradients for planning.
Revised: As we experimentally show, this not only leads to models that perform better in their prediction accuracy compared to models learned on top of point-cloud or occupancy object representations, but SDFs also enable the functionals/learned models to have more useful gradients for planning.

Original: In the experiments, we demonstrate the versatility of our approach by tackling two completely different tasks within one framework:
Revised: In the experiments, we demonstrate the versatility of our approach by tackling two completely different tasks within one framework:

Original: On the one hand, a kinematic task where the goal is to hang mugs of different shapes on hooks of different shapes.
Revised: On the one hand, a kinematic task where the goal is to hang mugs of different shapes on hooks of different shapes.

Original: On the other hand, a pushing scenario where boxes and L-shaped objects should be pushed to different goal regions by pushers of different sizes.
Revised: On the other hand, a pushing scenario where boxes and L-shaped objects should be pushed to different goal regions by pushers of different sizes.

Original: In the ﬁrst case, the model predicts whether the static interaction between SDFs leads to manipulation success, whereas in the latter case, the model predicts the forward dynamics in SDF space based on a history of SDF interactions of two objects.
Revised: In the ﬁrst case, the model predicts whether the static interaction between SDFs leads to manipulation success, whereas in the latter case, the model predicts the forward dynamics in SDF space based on a history of SDF interactions of two objects.

Original: We show that our framework can be used to plan motions that involve multiple push phases.
Revised: We show that our framework can be used to plan motions that involve multiple push phases.

Original: To summarize our main contributions, we propose
Revised: To summarize our main contributions, we propose

Original: • To learn a novel class of kinematic and dynamic models as functionals of SDFs, • A manipulation planning framework that utilizes these learned functionals as constraints, • Comparison to other object representations showing the advantages of the SDFs.
Revised: • To learn a novel class of kinematic and dynamic models as functionals of SDFs, • A manipulation planning framework that utilizes these learned functionals as constraints, • Comparison to other object representations showing the advantages of the SDFs.

Original:  Related Work2.1 Signed Distance Fields as Object Representation
Revised:  Related Work

Revised: 2.1 Signed Distance Fields as Object Representation

Original: Representing objects or scenes as implicit surfaces [15, 16, 17] or SDFs [18, 19, 20, 21, 22, 23] is an active research topic, due to aforementioned advantages like learning shape completion, non-convex shapes etc.
Revised: Representing objects or scenes as implicit surfaces [15, 16, 17] or SDFs [18, 19, 20, 21, 22, 23] is an active research topic, due to aforementioned advantages like learning shape completion, non-convex shapes etc.

Original: Our focus is not to obtain SDFs from observations in the ﬁrst place.
Revised: Our focus is not to obtain SDFs from observations in the ﬁrst place.

Original: Conversely, we are interested in what can be done with SDFs in the context of model learning and manipulation planning.
Revised: Conversely, we are interested in what can be done with SDFs in the context of model learning and manipulation planning.

Original: There are some works that utilize SDFs within trajectory optimization [24, 25, 26], but without learning or integration into a TAMP framework.
Revised: There are some works that utilize SDFs within trajectory optimization [24, 25, 26], but without learning or integration into a TAMP framework.

Original: While some recent approaches [27, 28, 29] have suggested that grasping of diverse objects can be addressed using implicit functions, we present a manipulation framework that utilizes SDFs for learning and formulating more general models.
Revised: While some recent approaches [27, 28, 29] have suggested that grasping of diverse objects can be addressed using implicit functions, we present a manipulation framework that utilizes SDFs for learning and formulating more general models.

Original: 2.2 Perceptual Models
Revised: 2.2 Perceptual Models

Original: There is great interest in learning predictive models in perception spaces, especially applied to the problem of pushing.
Revised: There is great interest in learning predictive models in perception spaces, especially applied to the problem of pushing.

Original: So-called visual foresight approaches [30, 31, 32] aim to predict the evolution of the scene in image space.
Revised: So-called visual foresight approaches [30, 31, 32] aim to predict the evolution of the scene in image space.

Original: Our SDF dynamics model is also closely related to perception spaces, but, in comparison, is naturally differentiable.
Revised: Our SDF dynamics model is also closely related to perception spaces, but, in comparison, is naturally differentiable.

Original: Xu et al.
Revised: Xu et al.

Original: [33] use a voxelized SDF-based representation of the whole scene to predict the motion of an object when an action is applied.
Revised: [33] use a voxelized SDF-based representation of the whole scene to predict the motion of an object when an action is applied.

Original: Our approach is more structured in the sense that we do not predict the scene ﬂow for actions applied on a single object, but the dynamics of interacting of objects.
Revised: Our approach is more structured in the sense that we do not predict the scene ﬂow for actions applied on a single object, but the dynamics of interacting of objects.

Original: In [34], the pushing dynamics in keypoints extracted from visual object observations is learned.
Revised: In [34], the pushing dynamics in keypoints extracted from visual object observations is learned.

Original: However, their focus is to utilize the learned model to stabilize a trajectory with MPC.
Revised: However, their focus is to utilize the learned model to stabilize a trajectory with MPC.

Original: We focus on planning a complex pushing trajectory and not stabilization during execution.
Revised: We focus on planning a complex pushing trajectory and not stabilization during execution.

Original: SE3 networks [35] learn a forward model that predicts a rigid transformation of an observed point cloud given actions.
Revised: SE3 networks [35] learn a forward model that predicts a rigid transformation of an observed point cloud given actions.

Original: However, they need ground-truth transformations at training time (we only need SDF observations).
Revised: However, they need ground-truth transformations at training time (we only need SDF observations).

Original: Where most of these approaches differ from our approach is that they assume the model to be a function of the observation of a single object or the scene and an action as input.
Revised: Where most of these approaches differ from our approach is that they assume the model to be a function of the observation of a single object or the scene and an action as input.

Original: Therefore, these approaches are mostly limited to the same pusher geometry and make the assumption that actions can readily be applied to the object.
Revised: Therefore, these approaches are mostly limited to the same pusher geometry and make the assumption that actions can readily be applied to the object.

Original: Our model handles the interaction between objects of different shapes and can plan the contact establishment phase as well.
Revised: Our model handles the interaction between objects of different shapes and can plan the contact establishment phase as well.

Original: Transporter networks [36] or deep visual reasoning [7] predict manipulation sequences from image spaces.
Revised: Transporter networks [36] or deep visual reasoning [7] predict manipulation sequences from image spaces.

Original: However, no dynamic models are considered in these approaches.
Revised: However, no dynamic models are considered in these approaches.

Original: 2.3 Manipulation Planning (with learned models)
Revised: 2.3 Manipulation Planning (with learned models)

Original: In [37, 38], a manipulation framework based on point cloud observations and manipulation primitives is proposed.
Revised: In [37, 38], a manipulation framework based on point cloud observations and manipulation primitives is proposed.

Original: Our method plans the complete motions based on learned dynamic models.
Revised: Our method plans the complete motions based on learned dynamic models.

Original: Sutanto et al.
Revised: Sutanto et al.

Original: [39] is related to our formulation in the sense that they learn manifolds that are used as constraints in sequential manipulation problems.
Revised: [39] is related to our formulation in the sense that they learn manifolds that are used as constraints in sequential manipulation problems.

Original: However, there are no dynamic models or dependencies on the geometry of the involved objects in the learned constraints.
Revised: However, there are no dynamic models or dependencies on the geometry of the involved objects in the learned constraints.

Original: You et al.
Revised: You et al.

Original: [40] address a hanging task similar to our mug hanging experiment on a more diverse set of object categories.
Revised: [40] address a hanging task similar to our mug hanging experiment on a more diverse set of object categories.

Original: They use a point-cloud-based input representation to predict a hanging pose.
Revised: They use a point-cloud-based input representation to predict a hanging pose.

Original: Therefore, they need a special neural network for collision avoidance (similar to [41]), while our SDF based representation can handle collisions directly.
Revised: Therefore, they need a special neural network for collision avoidance (similar to [41]), while our SDF based representation can handle collisions directly.

Original: Further, we learn a manifold of solutions instead of predicting a single hanging conﬁguration.
Revised: Further, we learn a manifold of solutions instead of predicting a single hanging conﬁguration.

Original: To summarize, what makes our approach unique is that we propose to use
Revised: To summarize, what makes our approach unique is that we propose to use

Original: SDFs as a common object representation that is closely connected to perception to learn a variety of models that are able to take the interaction of objects into account and can be integrated in an optimization-based motion planning framework due to their differentiability.
Revised: SDFs as a common object representation that is closely connected to perception to learn a variety of models that are able to take the interaction of objects into account and can be integrated in an optimization-based motion planning framework due to their differentiability.

Original:  Background on Signed-Distance Fields (SDFs)
Revised:  Background on Signed-Distance Fields (SDFs)

Original: Let Ω ⊂ R 3 be an object in the 3D Euclidean space.
Revised: Let Ω ⊂ R 3 be an object in the 3D Euclidean space.

Original: A function φ : R 3 → R , φ ∈ Φ with φ ( x ) = − d ( x, ∂ Ω) for x ∈ Ω and φ ( x ) =
Revised: A function φ : R 3 → R , φ ∈ Φ with φ ( x ) = − d ( x, ∂ Ω) for x ∈ Ω and φ ( x ) =

Original: d ( x, ∂ Ω) for x ∈ R 3 \ Ω is called a signed-distance ﬁeld of Ω in R 3 .
Revised: d ( x, ∂ Ω) for x ∈ R 3 \ Ω is called a signed-distance ﬁeld of Ω in R 3 .

Original: Here, d ( x, ∂ Ω) =
Revised: Here, d ( x, ∂ Ω) =

Original: inf x (cid:

Revised: inf x (cid:

Original: ) ∈
Revised: ) ∈

Original: ∂ Ω (cid:107) x − x (cid:48) (cid:107) 2 and ∂ Ω the boundary of Ω .
Revised: ∂ Ω (cid:107) x − x (cid:48) (cid:107) 2 and ∂ Ω the boundary of Ω .

Original: We assume φ to be differentiable almost everywhere in R 3 .
Revised: We assume φ to be differentiable almost everywhere in R 3 .

Original: The way φ is deﬁned ensures that inside the object, φ attains negative values, on the boundary zeros, and outside positive ones.
Revised: The way φ is deﬁned ensures that inside the object, φ attains negative values, on the boundary zeros, and outside positive ones.

Original: We denote with the set Φ the space of all functions φ that are SDFs for some object.
Revised: We denote with the set Φ the space of all functions φ that are SDFs for some object.

Original: Rigid Transformations of SDFs A central concept in this work is to rigidly transform SDFs in space.
Revised: Rigid Transformations of SDFs A central concept in this work is to rigidly transform SDFs in space.

Original: This can be realized by transforming the input where the SDF is queried.
Revised: This can be realized by transforming the input where the SDF is queried.

Original: To simplify the notation, we deﬁne a rigid transformation , parameterized by q ∈ R 7 (translation + quaternion),
Revised: To simplify the notation, we deﬁne a rigid transformation , parameterized by q ∈ R 7 (translation + quaternion),

Original: [Equation]
Revised: [Equation]

Original: of an SDF φ , where R ( q ) ∈ R 3 × 3 is a rotation matrix and r ( q ) ∈ R 3 the translation vector.
Revised: of an SDF φ , where R ( q ) ∈ R 3 × 3 is a rotation matrix and r ( q ) ∈ R 3 the translation vector.

Original:  Manipulation Planning with Signed-Distance Functionals
Revised:  Manipulation Planning with Signed-Distance Functionals

Original: The core idea of this work is to represent each object i in the scene as a signed-distance ﬁeld φ i in order to learn predictive models as functionals H of these SDFs.
Revised: The core idea of this work is to represent each object i in the scene as a signed-distance ﬁeld φ i in order to learn predictive models as functionals H of these SDFs.

Original: Based on the learned functionals, we formulate a trajectory optimization problem where the decision variable is a trajectory of rigid transformations applied on the initial SDFs as they have been observed in the initial scene.
Revised: Based on the learned functionals, we formulate a trajectory optimization problem where the decision variable is a trajectory of rigid transformations applied on the initial SDFs as they have been observed in the initial scene.

Original: More speciﬁcally, through interaction with the environment, we aim to learn functionals of the form H : Φ × · · · × Φ → R that map multiple SDFs of multiple, possible different objects at possibly different consecutive times to a real number.
Revised: More speciﬁcally, through interaction with the environment, we aim to learn functionals of the form H : Φ × · · · × Φ → R that map multiple SDFs of multiple, possible different objects at possibly different consecutive times to a real number.

Original: These are trained in a way that a value of zero implies that the SDFs as input are compatible with what has been learned through interaction with the environment.
Revised: These are trained in a way that a value of zero implies that the SDFs as input are compatible with what has been learned through interaction with the environment.

Original: Otherwise, they should attain a positive value, hence functionals H discriminate correct from incorrect dynamics or desired from undesired manipulations.
Revised: Otherwise, they should attain a positive value, hence functionals H discriminate correct from incorrect dynamics or desired from undesired manipulations.

Original: The learned functionals then deﬁne constraints for the (hybrid) trajectory optimization problem
Revised: The learned functionals then deﬁne constraints for the (hybrid) trajectory optimization problem

Original: [Equation]
Revised: [Equation]

Original: The discrete variable s k determines which functionals H from the set H ( s k ( t ) ) are active at which of the K ∈ N phases of the motion ( k ( t )
Revised: The discrete variable s k determines which functionals H from the set H ( s k ( t ) ) are active at which of the K ∈ N phases of the motion ( k ( t )

Original: = (cid:98) t/T (cid:99) ).
Revised: = (cid:98) t/T (cid:99) ).

Original: This number of phases is part of the decision problem.
Revised: This number of phases is part of the decision problem.

Original: The trajectory q 0: KT of rigid transformations is discretized in time into T ∈ N steps per phase.
Revised: The trajectory q 0: KT of rigid transformations is discretized in time into T ∈ N steps per phase.

Original: If n O is the number of objects in the scene S , then q t ∈ R 7 · n O , leading to 7 · KTn O continuous variables.
Revised: If n O is the number of objects in the scene S , then q t ∈ R 7 · n O , leading to 7 · KTn O continuous variables.

Original: Further, s k selects through the set I H ( s k ( t ) ) the time slice and object index tuples ( t, i ) that determine the SDFs φ i , which have been transformed through q i t , at the times t of the trajectory on which the functional constraints H depends on.
Revised: Further, s k selects through the set I H ( s k ( t ) ) the time slice and object index tuples ( t, i ) that determine the SDFs φ i , which have been transformed through q it , at the times t of the trajectory on which the functional constraints H depends on.

Original: This problem formulation is inspired by LGP [10], but the constraints are replaced by learned functionals of SDFs.
Revised: This problem formulation is inspired by LGP [10], but the constraints are replaced by learned functionals of SDFs.

Original: The set S ( S ) contains all valid sequences s 1: K of such discrete variables for the scene S .
Revised: The set S ( S ) contains all valid sequences s 1: K of such discrete variables for the scene S .

Original: The goal of the manipulation problem is speciﬁed through S ( S ) by s K selecting a desired goal functional constraint that has to be fulﬁlled at the end q KT of the trajectory.
Revised: The goal of the manipulation problem is speciﬁed through S ( S ) by s K selecting a desired goal functional constraint that has to be fulﬁlled at the end q KT of the trajectory.

Original: Solving (2) therefore involves a tree search over nodes s 1: K such that the continuous optimization problem implied by the choice of s 1: K at a node of the tree is feasible.
Revised: Solving (2) therefore involves a tree search over nodes s 1: K such that the continuous optimization problem implied by the choice of s 1: K at a node of the tree is feasible.

Original: The role of q in the optimization problem is not absolute object poses, but rather rigid transformations applied to the SDFs φ i that represent the conﬁgurations of the objects as observed in the scene initially.
Revised: The role of q in the optimization problem is not absolute object poses, but rather rigid transformations applied to the SDFs φ i that represent the conﬁgurations of the objects as observed in the scene initially.

Original: With the term c , we can include regularizing motion costs.
Revised: With the term c , we can include regularizing motion costs.

Original: As will be described in sec.
Revised: As will be described in sec.

Original: 5.1, the forward dynamic model we learn for pushing implies a constraint on the evolution of one object based on the motion of another object.
Revised: 5.1, the forward dynamic model we learn for pushing implies a constraint on the evolution of one object based on the motion of another object.

Original: Therefore, we only add motion costs to those degrees of freedom that can be interpreted as being controlled, meaning the motion of the other object.
Revised: Therefore, we only add motion costs to those degrees of freedom that can be interpreted as being controlled, meaning the motion of the other object.

Original: From the perspective of (2), there is no explicit notion of controlled actions.
Revised: From the perspective of (2), there is no explicit notion of controlled actions.

Original:  Deep Signed-Distance Functionals
Revised:  Deep Signed-Distance Functionals

Original: This section presents two main types of models we propose.
Revised: This section presents two main types of models we propose.

Original: First, a way of learning forward dynamic models that predict the dynamics in SDF space based on the interaction between objects.
Revised: First, a way of learning forward dynamic models that predict the dynamics in SDF space based on the interaction between objects.

Original: Second, a kinematic success model that determines whether a static conﬁguration of interacting SDFs leads to manipulation success.
Revised: Second, a kinematic success model that determines whether a static conﬁguration of interacting SDFs leads to manipulation success.

Original: All functionals we consider are of the form H : Φ × · · · × Φ → R , i.e. they only take the SDFs of interacting objects as input, there is no explicit notion of position, orientation, action etc.
Revised: All functionals we consider are of the form H : Φ × · · · × Φ → R , i.e. they only take the SDFs of interacting objects as input, there is no explicit notion of position, orientation, action etc.

Original: Therefore, the functionals can be used at arbitrary locations in space.
Revised: Therefore, the functionals can be used at arbitrary locations in space.

Original: Bounding-Box To deﬁne most of the following functionals and those in sec.
Revised: Bounding-Box To deﬁne most of the following functionals and those in sec.

Original: 6, we utilize a set
Revised: 6, we utilize a set

Original: X with the property Ω ⊂ X ⊂ R 3 for all objects Ω that are involved.
Revised: X with the property Ω ⊂ X ⊂ R 3 for all objects Ω that are involved.

Original: This set should be large enough to cover the relevant workspace of the manipulation problem where the interaction between the objects should occur.
Revised: This set should be large enough to cover the relevant workspace of the manipulation problem where the interaction between the objects should occur.

Original: A more detailed discussion about the role of X can be found in sec.
Revised: A more detailed discussion about the role of X can be found in sec.

Original: 5.1 Forward Dynamic Models
Revised: 5.1 Forward Dynamic Models

Original: Generally, a forward model predicts future states/observations of a system given the current or additionally a history of states/observations.
Revised: Generally, a forward model predicts future states/observations of a system given the current or additionally a history of states/observations.

Original: In the context of objects being represented solely as SDFs, we propose to learn a forward model F : Φ × · · · × Φ → Φ that predicts the SDF of an object φ 1 t at time step t based on a history of SDF observations of the object φ 1 t − l : t − 1 until time t − 1 and the motion of another object φ 2 t − l : t until time t .
Revised: In the context of objects being represented solely as SDFs, we propose to learn a forward model F : Φ × · · · × Φ → Φ that predicts the SDF of an object φ 1 t at time step t based on a history of SDF observations of the object φ 1 t − l : t − 1 until time t − 1 and the motion of another object φ 2 t − l : t until time t .

Original: This means F as
Revised: This means F as

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  is an SDF itself that can be queried in R 3 .
Revised: ) is an SDF itself that can be queried in R 3 .

Original: Interactions between more than two objects are possible, but we focus on pair-interactions in the present work.
Revised: Interactions between more than two objects are possible, but we focus on pair-interactions in the present work.

Original: If l = 1 , F is a quasi-static model.
Revised: If l = 1 , F is a quasi-static model.

Original: Internally,
Revised: Internally,

Original: F can be deﬁned to either directly predict the SDF φ 1 t as in (3) or the ﬂow
Revised: F can be deﬁned to either directly predict the SDF φ 1 t as in (3) or the ﬂow

Original: [Equation]
Revised: [Equation]

Original:  from φ 1 t − 1 to φ 1 t .
Revised: (4) from φ 1 t − 1 to φ 1 t .

Original: In both cases, the functional H for planning is then naturally deﬁned as
Revised: In both cases, the functional H for planning is then naturally deﬁned as

Original: [Equation]
Revised: [Equation]

Original: For a perfect model F , this functional H F attains a zero value if and only if the evolution of φ 1 t − l [Equation] t and φ 2 t − l : t is compatible with the underlying physical process in the space X .
Revised: For a perfect model F , this functional H F attains a zero value if and only if the evolution of φ 1 t − l : t and φ 2 t − l : t is compatible with the underlying physical process in the space X .

Original: Therefore, the loss function to train F is also (5) for a dataset D = (cid:8)(cid:0) φ 10: l , φ 20: l (cid:1) i (cid:9) n i =1 of such consecutive SDF motions of the two objects.
Revised: Therefore, the loss function to train F is also (5) for a dataset D = (cid:8)(cid:0) φ 10: l , φ 20: l (cid:1) i (cid:9) n i =1 of such consecutive SDF motions of the two objects.

Original: Since F takes as input the complete SDFs of the objects and not just values like the distance between objects and their contact point locations, it can learn to reason not only about these quantities, but also the contact geometry, relative object movements, center of mass and inertial parameters (assuming an equal density of the objects), all of which are necessary quantities to represent the dynamics.
Revised: Since F takes as input the complete SDFs of the objects and not just values like the distance between objects and their contact point locations, it can learn to reason not only about these quantities, but also the contact geometry, relative object movements, center of mass and inertial parameters (assuming an equal density of the objects), all of which are necessary quantities to represent the dynamics.

Original: This way, F inherently takes the geometry of the objects into account.
Revised: This way, F inherently takes the geometry of the objects into account.

Original: Note that usually, forward models are understood in terms of a function that maps the current state (history) and an (abstract) action a to the next state.
Revised: Note that usually, forward models are understood in terms of a function that maps the current state (history) and an (abstract) action a to the next state.

Original: For SDFs, this would mean a model of the form φ 1 t = F [ φ 1 t − l : t − 1 , a t − 1 ] .
Revised: For SDFs, this would mean a model of the form φ 1 t = F [ φ 1 t − l : t − 1 , a t − 1 ] .

Original: In our case, however, there is no notion of an abstract action, instead, our formulation learns a generic model of the interaction between two objects, where the motion of one object ( φ 2 ) inﬂuences the other ( φ 1 ).
Revised: In our case, however, there is no notion of an abstract action, instead, our formulation learns a generic model of the interaction between two objects, where the motion of one object ( φ 2 ) inﬂuences the other ( φ 1 ).

Original: Therefore, while the transformation applied to φcan be interpreted as an action, the model has no action as input and hence can deal with different geometries of φ 2 , which is not possible in case of an abstract action without φ 2 also being an input.
Revised: Therefore, while the transformation applied to φcan be interpreted as an action, the model has no action as input and hence can deal with different geometries of φ 2 , which is not possible in case of an abstract action without φ 2 also being an input.

Original: 5.2 Kinematic Success Models
Revised: 5.2 Kinematic Success Models

Original: Many tasks in manipulation planning can be speciﬁed in terms of static success models instead of a full forward dynamics model.
Revised: Many tasks in manipulation planning can be speciﬁed in terms of static success models instead of a full forward dynamics model.

Original: We call a model that predicts whether a conﬁguration of potentially multiple SDFs at the same time slice leads to manipulation success a kinematic success model.
Revised: We call a model that predicts whether a conﬁguration of potentially multiple SDFs at the same time slice leads to manipulation success a kinematic success model.

Original: Assume through interaction with the environment, a dataset D = (cid:8) ( φ j ) j ∈I i , y i (cid:9) n i =1 of SDFs representing |I
Revised: Assume through interaction with the environment, a dataset D = (cid:8) ( φ j ) j ∈I i , y i (cid:9) n i =1 of SDFs representing |I

Original: i | many objects has been obtained with y i =
Revised: i | many objects has been obtained with y i =

Original: 1 indicating that the conﬁguration of SDFs leads to manipulation success, y i = 0 to failure.
Revised: 1 indicating that the conﬁguration of SDFs leads to manipulation success, y i = 0 to failure.

Original: Then learning H is similar to a classiﬁcation problem, where H (cid:0) ( φ i ) i ∈I (cid:1) = 0 implies success prediction.
Revised: Then learning H is similar to a classiﬁcation problem, where H (cid:0) ( φ i ) i ∈I (cid:1) = 0 implies success prediction.

Original: This way, H can model a manifold of feasible conﬁgurations and not only a single solution.
Revised: This way, H can model a manifold of feasible conﬁgurations and not only a single solution.

Original: See sec.
Revised: See sec.

Original: F for details (loss function etc.).
Revised: F for details (loss function etc.).

Original: 5.3 Learning Functionals with Neural Networks
Revised: 5.3 Learning Functionals with Neural Networks

Original: So far, we have not discussed how functionals of the form H : Φ × · · · × Φ → R can be learned or even queried in the ﬁrst place with usual function approximators like neural networks, since, in general, the neural network would have to take functions as inﬁnite dimensional objects as input.
Revised: So far, we have not discussed how functionals of the form H : Φ × · · · × Φ → R can be learned or even queried in the ﬁrst place with usual function approximators like neural networks, since, in general, the neural network would have to take functions as inﬁnite dimensional objects as input.

Original: To approximate this, we choose in this work the straight-forward approach by evaluating φ ∈ Φ on a discretized version of the set X , denoted by X h .
Revised: To approximate this, we choose in this work the straight-forward approach by evaluating φ ∈ Φ on a discretized version of the set X , denoted by X h .

Original: As discussed previously, the set X should cover the relevant region of the workspace where the interaction between the objects takes place.
Revised: As discussed previously, the set X should cover the relevant region of the workspace where the interaction between the objects takes place.

Original: We speciﬁcally do not assume X to be aligned or perfectly centered with the objects that are involved.
Revised: We speciﬁcally do not assume X to be aligned or perfectly centered with the objects that are involved.

Original: This way, the dynamics model from sec.
Revised: This way, the dynamics model from sec.

Original: 5.1 can be realized by
Revised: 5.1 can be realized by

Original: [Equation]
Revised: [Equation]

Original: (6) with F θ being usual neural network architectures.
Revised: with F θ being usual neural network architectures.

Original: During training, the integral in (5) is approximated over the same discretized X h for simplicity.
Revised: During training, the integral in (5) is approximated over the same discretized X h for simplicity.

Original: Hence, the dataset to train F can contain the SDF observations at the grid points of X h only.
Revised: Hence, the dataset to train F can contain the SDF observations at the grid points of X h only.

Original: However, F θ still approximates an SDF which can be queried at arbitrary x ∈ R 3 and does not only predict the values on the grid points.
Revised: However, F θ still approximates an SDF which can be queried at arbitrary x ∈ R 3 and does not only predict the values on the grid points.

Original: For general functionals H , the evaluation is analogous, i.e. H (cid:16)(cid:0) φ i (cid:1) i ∈I (cid:17)
Revised: For general functionals H , the evaluation is analogous, i.e. H (cid:16)(cid:0) φ i (cid:1) i ∈I (cid:17)

Original: ≈ H θ (cid:16)(cid:0) φ
Revised: ≈ H θ (cid:16)(cid:0) φ

Original: i ( X h ) (cid:1) i ∈I (cid:17) .
Revised: i ( X h ) (cid:1) i ∈I (cid:17) .

Original: Technically, X h ⊂ R d × h × w is a regular grid which allows us to encode φ ( X h ) using 2D or 3D convolutions.
Revised: Technically, X h ⊂ R d × h × w is a regular grid which allows us to encode φ ( X h ) using 2D or 3D convolutions.

Original: In contrast to an occupancy grid, the evaluation of φ ( X h ) contains more information about the object than whether there is an object at the grid point or not.
Revised: In contrast to an occupancy grid, the evaluation of φ ( X h ) contains more information about the object than whether there is an object at the grid point or not.

Original: Note that the differentiabilty of H (cid:16)(cid:0) T ( q i )[ φ i ( X h )] (cid:1) i ∈I (cid:17) with respect to q i is maintained, which is another advantage of representing such models as functionals of SDF functions evaluated on a grid instead of static values on a grid.
Revised: Note that the differentiabilty of H (cid:16)(cid:0) T ( q i )[ φ i ( X h )] (cid:1) i ∈I (cid:17) with respect to q i is maintained, which is another advantage of representing such models as functionals of SDF functions evaluated on a grid instead of static values on a grid.

Original: During training, it is sufﬁcient to only have the SDF values evaluated on a gird, no other information like actions or velocity/pose estimations are needed.
Revised: During training, it is sufﬁcient to only have the SDF values evaluated on a gird, no other information like actions or velocity/pose estimations are needed.

Original:  Task Constraint Functionals
Revised:  Task Constraint Functionals

Original: Here we present analytical functionals of SDFs that are useful to specify goals of a manipulation problem or other task aspects.
Revised: Here we present analytical functionals of SDFs that are useful to specify goals of a manipulation problem or other task aspects.

Original: These functionals are general as a direct consequence of our object representations being SDFs.
Revised: These functionals are general as a direct consequence of our object representations being SDFs.

Original: Therefore, there is no advantage or need to learn these given the SDFs.
Revised: Therefore, there is no advantage or need to learn these given the SDFs.

Original: 6.1 Pair-Collision between Objects
Revised: 6.1 Pair-Collision between Objects

Original: Collision avoidance is an inherent part of many task speciﬁcations.
Revised: Collision avoidance is an inherent part of many task speciﬁcations.

Original: Given two SDFs φ 1 , φ 2 , we can measure whether they are in collision via their overlap integral
Revised: Given two SDFs φ 1 , φ 2 , we can measure whether they are in collision via their overlap integral

Original: [Equation]
Revised: [Equation]

Original: The indicator bracket [ · ] means [ P ] = 

Revised: The indicator bracket [ · ] means [ P ] =

Original: if P is true, otherwise [ P ] = 0 .
Revised: if P is true, otherwise [ P ] = 0 .

Original: The integral in (7) integrates over the space where both SDFs are negative at the same time, which is only the case if the two objects overlap, hence are in collision.
Revised: The integral in (7) integrates over the space where both SDFs are negative at the same time, which is only the case if the two objects overlap, hence are in collision.

Original: The gradients of (7) are smoothed using the sigmoid function σ ( z ) =
Revised: The gradients of (7) are smoothed using the sigmoid function σ ( z ) =

Original: [Equation]
Revised: [Equation]

Revised: +exp

Original: [Equation]
Revised: [Equation]

Original: a
Revised: , i.e. [ [Equation] = [Equation]

Original: [Equation]
Revised: [Equation]

Revised: with a parameter a > 0 .

Original: 6.2 Goal Region
Revised: 6.2 Goal Region

Original: If part of the task speciﬁcation is that an object φ 1 is fully contained inside the boundary of another object φ g , called the goal region, then a similar integral as for the pair-collision can be utilized
Revised: If part of the task speciﬁcation is that an object φ 1 is fully contained inside the boundary of another object φ g , called the goal region, then a similar integral as for the pair-collision can be utilized

Original: [Equation]
Revised: [Equation]

Original: Here, points outside of the goal region that are inside the object count towards the integral.
Revised: Here, points outside of the goal region that are inside the object count towards the integral.

Original: [Table]
Revised: [Table]

Original: 6.3 Establishing Contact between Objects
Revised: 6.3 Establishing Contact between Objects

Original: Establishing and maintaining contact between objects is central for many manipulation tasks.
Revised: Establishing and maintaining contact between objects is central for many manipulation tasks.

Original: One way to model that the distance between two objects φ 1 and φ 2 should be zero is via the functional
Revised: One way to model that the distance between two objects φ 1 and φ 2 should be zero is via the functional

Original: [Equation]
Revised: [Equation]

Original:  Experiments 7.1 Mug-Hanging: Kinematic Success Model
Revised:  Experiments 7.1 Mug-Hanging: Kinematic Success Model

Original: In this experiment we want to ﬁnd rigid transformations applied on observed mugs of different shapes in a scene to hang them stably on hooks of different types.
Revised: In this experiment we want to ﬁnd rigid transformations applied on observed mugs of different shapes in a scene to hang them stably on hooks of different types.

Original: The functional H hang is therefore a kinematic success model that takes the SDFs of the mug and the hook as input.
Revised: The functional H hang is therefore a kinematic success model that takes the SDFs of the mug and the hook as input.

Original: To generate data to learn H hang , we randomly sample scenes of different mug and hook shapes (1600 scenes for training, 400 for testing and 150 for evaluation).
Revised: To generate data to learn H hang , we randomly sample scenes of different mug and hook shapes (1600 scenes for training, 400 for testing and 150 for evaluation).

Original: See Fig.
Revised: See Fig.

Original: 15 for examples of mugs and hooks in the evaluation data.
Revised: 15 for examples of mugs and hooks in the evaluation data.

Original: Then we sample for each scene in the training and test data the position and orientation of the mug uniformly in the bounding box X until at least one successful conﬁguration has been obtained where the mug does not fall on the ground when being dropped from the sampled conﬁguration while at the same time not being in collision with the hook.
Revised: Then we sample for each scene in the training and test data the position and orientation of the mug uniformly in the bounding box X until at least one successful conﬁguration has been obtained where the mug does not fall on the ground when being dropped from the sampled conﬁguration while at the same time not being in collision with the hook.

Original: We use Bullet [42] to simulate the dropping.
Revised: We use Bullet [42] to simulate the dropping.

Original: In total, 20 conﬁgurations per scene are generated.
Revised: In total, 20 conﬁgurations per scene are generated.

Original: Since sampling a successful conﬁguration is a rare event, for the majority of the scenes, only one successful and 19 failure conﬁgurations are contained in the training and test data, making learning challenging.
Revised: Since sampling a successful conﬁguration is a rare event, for the majority of the scenes, only one successful and 19 failure conﬁgurations are contained in the training and test data, making learning challenging.

Original: Another challenge of this task is that the model has to reason about both the hook and mug geometry jointly.
Revised: Another challenge of this task is that the model has to reason about both the hook and mug geometry jointly.

Original: Formulating an analytical model, e.g. on a mesh-based object representation, to model this constraint is non-trivial.
Revised: Formulating an analytical model, e.g. on a mesh-based object representation, to model this constraint is non-trivial.

Original: 7.1.1 Performance with Optimization
Revised: 7.1.1 Performance with Optimization

Original: Fig.
Revised: Fig.

Original: 1 shows solution conﬁgurations found by our model H hang as an optimization objective.
Revised: 1 shows solution conﬁgurations found by our model H hang as an optimization objective.

Original: Interestingly, the solutions not always contain the intuitive solution, but also ones where other parts of the hook are being utilized (middle column in Fig. 1).
Revised: Interestingly, the solutions not always contain the intuitive solution, but also ones where other parts of the hook are being utilized (middle column in Fig. 1).

Original: The optimization problem (2) to solve this mug hanging problem has two objectives, the learned kinematic success functional H hang and the pair-collision H coll from sec. 6.1.
Revised: The optimization problem (2) to solve this mug hanging problem has two objectives, the learned kinematic success functional H hang and the pair-collision H coll from sec. 6.1.

Original: While in principle H hang also learns to avoid collisions, we found that the robustness in avoiding collisions increases when including H coll .
Revised: While in principle H hang also learns to avoid collisions, we found that the robustness in avoiding collisions increases when including H coll .

Original: The learned functional H hang is, in general, non-convex in the rigid transformation q of the mug.
Revised: The learned functional H hang is, in general, non-convex in the rigid transformation q of the mug.

Original: Therefore, we observed that gradient-based optimization is not sufﬁcient for the optimizer to ﬁnd a feasible solution, i.e. where H hang predicts zero, in every instance [43].
Revised: Therefore, we observed that gradient-based optimization is not sufﬁcient for the optimizer to ﬁnd a feasible solution, i.e. where H hang predicts zero, in every instance [43].

Original: To overcome this issue, we restart the optimization procedure up to 20 times with a randomly sampled initial guess of the mug in X .
Revised: To overcome this issue, we restart the optimization procedure up to 20 times with a randomly sampled initial guess of the mug in X .

Original: Fig.
Revised: Fig.

Original: 16 shows an example of a sampled initial conﬁguration from which the optimizer is started (left), then the optimized conﬁguration (middle) and ﬁnally, the conﬁguration after simulation.
Revised: 16 shows an example of a sampled initial conﬁguration from which the optimizer is started (left), then the optimized conﬁguration (middle) and ﬁnally, the conﬁguration after simulation.

Original: Tab. 1 shows the success rates on the evaluation scenes.
Revised: Tab. 1 shows the success rates on the evaluation scenes.

Original: As one can see, for the proposed approach where objects are represented as SDFs using optimization and sampling, in 98.7% of the evaluation scenes, a solution is found where H hang predicts success and no collision is violated (ﬁrst column).
Revised: As one can see, for the proposed approach where objects are represented as SDFs using optimization and sampling, in 98.7% of the evaluation scenes, a solution is found where H hang predicts success and no collision is violated (ﬁrst column).

Original: Out of these, 88.5% are stable conﬁgurations (checked by simulation) and the optimized conﬁguration of the mug is collision free with the hook, leading to 87.3% total solved scenes (last column).
Revised: Out of these, 88.5% are stable conﬁgurations (checked by simulation) and the optimized conﬁguration of the mug is collision free with the hook, leading to 87.3% total solved scenes (last column).

Original: When the optimization is run only once (second row), then only in 51.3% of the cases it converges to a feasible solution.
Revised: When the optimization is run only once (second row), then only in 51.3% of the cases it converges to a feasible solution.

Original: 7.1.2 Comparison to Sampling, Point-Cloud and Occupancy Measure Representations
Revised: 7.1.2 Comparison to Sampling, Point-Cloud and Occupancy Measure Representations

Original: This section shows that learning a kinematic success model based on the proposed SDF object representation outperforms other representations (point-clouds and occupancy measures) and further highlights the advantage of the models learned with SDFs providing useful gradients by comparing it to sampling without optimization.
Revised: This section shows that learning a kinematic success model based on the proposed SDF object representation outperforms other representations (point-clouds and occupancy measures) and further highlights the advantage of the models learned with SDFs providing useful gradients by comparing it to sampling without optimization.

Original: For full results, refer to sec.
Revised: For full results, refer to sec.

Original: A and Tab. 3.
Revised: A and Tab. 3.

Original: The sampling approach draws relative transformations of the mugs uniformly in X until the evaluation with the learned H hang and the collision functional H coll predicts a successful and collision free conﬁguration with a threshold κ .
Revised: The sampling approach draws relative transformations of the mugs uniformly in X until the evaluation with the learned H hang and the collision functional H coll predicts a successful and collision free conﬁguration with a threshold κ .

Original: As one can see in Tab. 1 and Tab. 3, our proposed approach has a signiﬁcantly higher performance (87.3%) compared to the best threshold for pure sampling with SDF (68.9%) and the best of the other object representations (34% with occupancy measure, 52% with point-cloud).
Revised: As one can see in Tab. 1 and Tab. 3, our proposed approach has a signiﬁcantly higher performance (87.3%) compared to the best threshold for pure sampling with SDF (68.9%) and the best of the other object representations (34% with occupancy measure, 52% with point-cloud).

Original: 7.2 Pushing Objects on a Table: Dynamic Model
Revised: 7.2 Pushing Objects on a Table: Dynamic Model

Original: In this experiment, we consider the task of pushing boxes and L-shaped objects of different dimensions with a spherical pusher of different radii into a goal region φ g on a table.
Revised: In this experiment, we consider the task of pushing boxes and L-shaped objects of different dimensions with a spherical pusher of different radii into a goal region φ g on a table.

Original: Fig.
Revised: Fig.

Original: 2 visualizes typical objects, pushers and goal regions.
Revised: 2 visualizes typical objects, pushers and goal regions.

Original: The goal region has to be large enough that all possible objects ﬁt.
Revised: The goal region has to be large enough that all possible objects ﬁt.

Original: We again use Bullet as a simulator to generate data to train a dynamics model of the from described in sec.
Revised: We again use Bullet as a simulator to generate data to train a dynamics model of the from described in sec.

Original: 5.1 with l = 1 , i.e. H F is a function of four SDFs φ 1 t , φ 1 t − 1 (object) and φ 2 t , φ 2 t − 1 (pusher).
Revised: 5.1 with l = 1 , i.e. H F is a function of four SDFs φ 1 t , φ 1 t − 1 (object) and φ 2 t , φ 2 t − 1 (pusher).

Original: In total, 14975 different scenes (including shapes and initial conﬁguration) are sampled where random push actions biased roughly towards the object center are applied until the object leaves the table.
Revised: In total, 14975 different scenes (including shapes and initial conﬁguration) are sampled where random push actions biased roughly towards the object center are applied until the object leaves the table.

Original: Since the dynamics and interaction of the objects in this scenario can be described in the 2D plane, the 3D signed distance functions of the objects are evaluated in the 2D set X h ∈ R 140 × 140 only.
Revised: Since the dynamics and interaction of the objects in this scenario can be described in the 2D plane, the 3D signed distance functions of the objects are evaluated in the 2D set X h ∈ R 140 × 140 only.

Original: Therefore, the model F predicts the dynamics of φ 1 in this 2D projection.
Revised: Therefore, the model F predicts the dynamics of φ 1 in this 2D projection.

Original: 7.2.1 Forward Prediction Error
Revised: 7.2.1 Forward Prediction Error

Original: [Table]
Revised: [Table]

Original: Tab. 2 shows the one-step prediction error on the evaluation dataset for the ﬂow model F ﬂow (4) and the direct
Revised: Tab. 2 shows the one-step prediction error on the evaluation dataset for the ﬂow model F ﬂow (4) and the direct

Original: SDF prediction F (3).
Revised: SDF prediction F (3).

Original: The way we utilize the model within the trajectory optimization problem never asks for predictions more than one step into the future.
Revised: The way we utilize the model within the trajectory optimization problem never asks for predictions more than one step into the future.

Original: We train one single dynamics model for both box and L-shaped objects and different pushers.
Revised: We train one single dynamics model for both box and L-shaped objects and different pushers.

Original: The prediction error is the RMSE of predicting the correct SDF values in X h .
Revised: The prediction error is the RMSE of predicting the correct SDF values in X h .

Original: The last row shows the error if the model would simply predict the next state as the last state of the object.
Revised: The last row shows the error if the model would simply predict the next state as the last state of the object.

Original: As one can see, F ﬂow achieves a lower error than F .
Revised: As one can see, F ﬂow achieves a lower error than F .

Original: This is due to F having to predict the complete SDF, while F ﬂow only the ﬂow.
Revised: This is due to F having to predict the complete SDF, while F ﬂow only the ﬂow.

Original: In phases of the motion where there is no contact between the object and the pusher, both models F ﬂow and F have to learn that the object should not move (and F has to predict the complete
Revised: In phases of the motion where there is no contact between the object and the pusher, both models F ﬂow and F have to learn that the object should not move (and F has to predict the complete

Original: SDF in this case as well), which is also non-trivial, but they accomplish this with low error.
Revised: SDF in this case as well), which is also non-trivial, but they accomplish this with low error.

Original: 7.2.2 Comparison to other Object Representations (Point-Cloud and Occupancy Measure)
Revised: 7.2.2 Comparison to other Object Representations (Point-Cloud and Occupancy Measure)

Original: In sec.
Revised: In sec.

Original: A.6.1 and sec.
Revised: A.6.1 and sec.

Original: A.6.2 we present and explain a comparison of the forward prediction error between models learned with object representations being SDFs, occupancy measures and pointclouds.
Revised: A.6.2 we present and explain a comparison of the forward prediction error between models learned with object representations being SDFs, occupancy measures and pointclouds.

Original: As shown in Tab. 4 and Tab. 5, models learned with the SDF representation outperform models based on point-clouds and occupancy measures in their predictive performance.
Revised: As shown in Tab. 4 and Tab. 5, models learned with the SDF representation outperform models based on point-clouds and occupancy measures in their predictive performance.

Original: Further, in
Revised: Further, in

Original: Tab. 5 and sec.
Revised: Tab. 5 and sec.

Original: A.4, we also show that one can learn image conditioned SDFs and dynamic models on top of the learned SDF simultaneously with no noticeable performance degradation.
Revised: A.4, we also show that one can learn image conditioned SDFs and dynamic models on top of the learned SDF simultaneously with no noticeable performance degradation.

Original: 7.2.3 Planning with the Learned Model and Execution Performance
Revised: 7.2.3 Planning with the Learned Model and Execution Performance

Original: Having learned the pushing dynamics prediction model, we now utilize it within (2) to solve the task of pushing the object into the goal region.
Revised: Having learned the pushing dynamics prediction model, we now utilize it within (2) to solve the task of pushing the object into the goal region.

Original: There are four constraints.
Revised: There are four constraints.

Original: First, the dynamics model H F and, second, the goal region H g .
Revised: First, the dynamics model H F and, second, the goal region H g .

Original: While this seems to be enough to specify the problem fully, we add two additional constraints, H coll and H PoC .
Revised: While this seems to be enough to specify the problem fully, we add two additional constraints, H coll and H PoC .

Original: The discrete variable s k of (2) decides whether there are one or two push phases.
Revised: The discrete variable s k of (2) decides whether there are one or two push phases.

Original: Only in a push phase, H PoC is active.
Revised: Only in a push phase, H PoC is active.

Original: H coll is always active.
Revised: H coll is always active.

Original: Similarly to the mug hanging experiment, local minima are a core issue as well.
Revised: Similarly to the mug hanging experiment, local minima are a core issue as well.

Original: Therefore, we initialize the pusher position at phase 1 or 2 on a set of 4 different points around the object.
Revised: Therefore, we initialize the pusher position at phase 1 or 2 on a set of 4 different points around the object.

Original: These 4 points around the object are always the same in all scenarios, no matter of the size, shape or orientation of the object.
Revised: These 4 points around the object are always the same in all scenarios, no matter of the size, shape or orientation of the object.

Original: Compared to other approaches where the action space has to be chosen much more carefully, we believe that this is a rather weak prior.
Revised: Compared to other approaches where the action space has to be chosen much more carefully, we believe that this is a rather weak prior.

Original: The initialization also does not start from contact with the object or similar, because our problem contains the challenge of contact establishment and possible breakage to push from a different side to achieve the goal.
Revised: The initialization also does not start from contact with the object or similar, because our problem contains the challenge of contact establishment and possible breakage to push from a different side to achieve the goal.

Original: Therefore, due to this initialization, there
Revised: Therefore, due to this initialization, there

Original: [Figure]
Revised: [Figure]

Original: a g
Revised: a g

Original: [Figure]
Revised: [Figure]

Original: i n
Revised: i n

Original: [Figure]
Revised: [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: are 20 different optimization problems we solve for each scene (4 for one pushing phase, 4 2 for two pushing phases).
Revised: are 20 different optimization problems we solve for each scene (4 for one pushing phase, 4 2 for two pushing phases).

Original: To evaluate the performance, we execute the planning result with the least constraint violation and cost open-loop in the simulator.
Revised: To evaluate the performance, we execute the planning result with the least constraint violation and cost open-loop in the simulator.

Original: Despite the fact that pushing is unstable over long-horizons, our proposed approach achieves a high performance.
Revised: Despite the fact that pushing is unstable over long-horizons, our proposed approach achieves a high performance.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 3, which plots the amount of the object that is inside of the goal region at the end of the execution, using the learned F ﬂow , the approach achieves 99.7% (median) coverage of the object inside the goal region on box pushing and 98.4% (median) on the L-shaped objects (50 evaluation scenes each).
Revised: 3, which plots the amount of the object that is inside of the goal region at the end of the execution, using the learned F ﬂow , the approach achieves 99.7% (median) coverage of the object inside the goal region on box pushing and 98.4% (median) on the L-shaped objects (50 evaluation scenes each).

Original: Please note that, although the goal region for small objects seems large, the optimizer usually moves the object until it is just barely inside the goal region and not any further.
Revised: Please note that, although the goal region for small objects seems large, the optimizer usually moves the object until it is just barely inside the goal region and not any further.

Original: Therefore, even very small deviations during the open-loop execution already lead to some parts sticking out.
Revised: Therefore, even very small deviations during the open-loop execution already lead to some parts sticking out.

Original: For the larger objects in the evaluation scenes, the goal region is barely large enough.
Revised: For the larger objects in the evaluation scenes, the goal region is barely large enough.

Original: With the direct F , the performance is a bit worse, but still high (median 96.6% for boxes, 91.5% for L-shaped objects).
Revised: With the direct F , the performance is a bit worse, but still high (median 96.6% for boxes, 91.5% for L-shaped objects).

Original: Sec.
Revised: Sec.

Original: B demonstrates that our proposed SDF framework outperforms an approach where the optimization problem is formulated for objects represented as meshes and the analytic dynamic model from [13] for the pushing dynamics.
Revised: B demonstrates that our proposed SDF framework outperforms an approach where the optimization problem is formulated for objects represented as meshes and the analytic dynamic model from [13] for the pushing dynamics.

Original: Refer to Fig.
Revised: Refer to Fig.

Original: 6 for quantitative results.
Revised: 6 for quantitative results.

Original: Finally, in sec.
Revised: Finally, in sec.

Original: A.6.
Revised: A.6.

Original: and Fig.
Revised: and Fig.

Original: 5 we investigate the importance of models learned on top providing useful gradients for planning in comparison to models with point-clouds and occupancy measures.
Revised: 5 we investigate the importance of models learned on top providing useful gradients for planning in comparison to models with point-clouds and occupancy measures.

Original: 7.2.4 Ablation Study
Revised: 7.2.4 Ablation Study

Original: Sec.
Revised: Sec.

Original: D presents an ablation study regarding the importance of the additional objectives H coll and H
Revised: D presents an ablation study regarding the importance of the additional objectives H coll and H

Original: PoC in the optimization problem.
Revised: PoC in the optimization problem.

Original: Results in Fig.
Revised: Results in Fig.

Original: 11 show that while it is possible to solve the scenes without them, the performance greatly increases if they are part of the problem formulation.
Revised: 11 show that while it is possible to solve the scenes without them, the performance greatly increases if they are part of the problem formulation.

Original: 7.2.5 Generalization to Out-of-Distribution Shapes, Multiple Objects, and Robots
Revised: 7.2.5 Generalization to Out-of-Distribution Shapes, Multiple Objects, and Robots

Original: In sec.
Revised: In sec.

Original: C.1, we demonstrate that the framework and the learned model generalizes to shapes beyond the training distribution.
Revised: C.1, we demonstrate that the framework and the learned model generalizes to shapes beyond the training distribution.

Original: See Fig.
Revised: See Fig.

Original: 2 and Fig.
Revised: 2 and Fig.

Original: 7 for those shapes.
Revised: 7 for those shapes.

Original: Quantitative results presented in
Revised: Quantitative results presented in

Original: Tab. 6 and Fig.
Revised: Tab. 6 and Fig.

Original: 8 indicate that the model achieves both high prediction accuracy and high performance when used for planning.
Revised: 8 indicate that the model achieves both high prediction accuracy and high performance when used for planning.

Original: Furthermore, as seen in Tab. 6, a model learned with SDFs generalizes signiﬁcantly better than with point-clouds, also relative to the results obtained on-distribution.
Revised: Furthermore, as seen in Tab. 6, a model learned with SDFs generalizes signiﬁcantly better than with point-clouds, also relative to the results obtained on-distribution.

Original: We further show in sec.
Revised: We further show in sec.

Original: C.3 and sec.
Revised: C.3 and sec.

Original: C.2 that the framework is capable of generalizing to scenes that contain obstacles (Fig. 10) and a scenario where three objects interact in order to solve the task (Fig. 9 and Fig. 4).
Revised: C.2 that the framework is capable of generalizing to scenes that contain obstacles (Fig. 10) and a scenario where three objects interact in order to solve the task (Fig. 9 and Fig. 4).

Original: Finally, in sec.
Revised: Finally, in sec.

Original: E we demonstrate multiple scenarios where the learned pushing dynamics model is embedded into a scene that contains robots.
Revised: E we demonstrate multiple scenarios where the learned pushing dynamics model is embedded into a scene that contains robots.

Original: All these generalization experiments require no change in the methodology or to learn a new model, showing the generality and versatility of our proposed framework.
Revised: All these generalization experiments require no change in the methodology or to learn a new model, showing the generality and versatility of our proposed framework.

Original: For more details, refer to the respective sections in the appendix.
Revised: For more details, refer to the respective sections in the appendix.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we have shown that the constraints of a trajectory optimization problem for solving manipulation problems can be formulated in terms of learned functionals of SDFs only.
Revised: In this work, we have shown that the constraints of a trajectory optimization problem for solving manipulation problems can be formulated in terms of learned functionals of SDFs only.

Original: SDFs can serve as a common object representation across completely different tasks.
Revised: SDFs can serve as a common object representation across completely different tasks.

Original: The functionals can naturally model the interaction between objects of arbitrary shapes and can be learned directly from
Revised: The functionals can naturally model the interaction between objects of arbitrary shapes and can be learned directly from

Original: SDF observations, which is closely connected to perception.
Revised: SDF observations, which is closely connected to perception.

Original: We have shown that learning models on top of SDFs outperform other object representations like point-clouds and occupancy measures both in terms of prediction accuracy and the ability to plan.
Revised: We have shown that learning models on top of SDFs outperform other object representations like point-clouds and occupancy measures both in terms of prediction accuracy and the ability to plan.

Original: The greatest challenge of our framework are local minima of the resulting trajectory optimization problem.
Revised: The greatest challenge of our framework are local minima of the resulting trajectory optimization problem.

Original: While sampling strategies for initial guesses can mitigate this to some extend, it is an issue, which is not unique to our approach, but many nonlinear trajectory optimization formulations.
Revised: While sampling strategies for initial guesses can mitigate this to some extend, it is an issue, which is not unique to our approach, but many nonlinear trajectory optimization formulations.

Original: While we have considered rigid objects in this work only, we believe that the proposed approach can be extended to deformables as well.
Revised: While we have considered rigid objects in this work only, we believe that the proposed approach can be extended to deformables as well.

Original: Acknowledgments
Revised: Acknowledgments

Original: Danny Driess thanks the International Max-Planck Research School for Intelligent Systems (IMPRS-IS) for the support.
Revised: Danny Driess thanks the International Max-Planck Research School for Intelligent Systems (IMPRS-IS) for the support.

Original: This research has been supported by the German Research Foundation (DFG) under Germany’s Excellence Strategy – EXC 2002/1–390523135 “Science of Intelligence”.
Revised: This research has been supported by the German Research Foundation (DFG) under Germany’s Excellence Strategy – EXC 2002/1–390523135 “Science of Intelligence”.

Original: The authors thank the anonymous reviewers for their comments.
Revised: The authors thank the anonymous reviewers for their comments.