Learning Visible Connectivity Dynamics for Cloth Smoothing
Anonymous Author(s)
Abstract: Robotic manipulation of cloth remains challenging due to the complex dynamics of cloth, lack of a low-dimensional state representation, and selfocclusions.
In contrast to previous model-based approaches that learn a pixelbased dynamics model or a compressed latent vector dynamics, we propose tolearn a particle-based dynamics model from a partial point cloud observation.
Toovercome the challenges of partial observability, we infer which visible points areconnected on the underlying cloth mesh.
We then learn a dynamics model over thisvisible connectivity graph.
Compared to previous learning-based approaches, ourmodel poses strong inductive bias with its particle based representation for learning the underlying cloth physics; it can generalize to cloths with novel shapes; it isinvariant to visual features; and the predictions can be more easily visualized.
Weshow that our method greatly outperforms previous state-of-the-art model-basedand model-free reinforcement learning methods in simulation.
Furthermore, wedemonstrate zero-shot sim-to-real transfer where we deploy the model trained insimulation on a Franka arm and show that the model can successfully smoothcloths of different materials, geometries and colors from crumpled conﬁgurations.
Videos can be found in the supplement and on our anonymous project website.
Introduction
Robotic manipulation of cloth has wide applications across both industrial and domestic tasks suchas laundry folding and bed making.
However, cloth manipulation remains challenging for roboticsdue to the complex cloth dynamics.
Further, like most deformable objects, cloth cannot be easilydescribed by low-dimensional state representations when placed in arbitrary conﬁgurations.
Selfocclusions make state estimation especially difﬁcult when the cloth is crumpled.
One approach to cloth manipulation explored by previous work, which we also adopt, is to learn acloth dynamics model and then use the model for planning to determine the robot actions.
However,given that a crumpled cloth has many self-occlusions and complex dynamics, it is unclear howto choose the appropriate state representation.
One possible state representation is to use a meshmodel of the entire cloth [1].
However, ﬁtting a full mesh model to an arbitrary crumpled clothconﬁguration is difﬁcult.
Recent work have approached fabric manipulation by either compressingthe cloth representation into a ﬁxed-size latent vector [2, 3, 4] or directly learning a visual dynamicsmodel in pixel space [5].
However, these representations do not enforce any inductive bias of thecloth physics, leading to suboptimal performance and generalization.
In contrast to a pixel-based or latent dynamics model, particle-based models have recently beenshown to be able to learn dynamics for ﬂuid and plastics [6, 7, 8].
A particle-based dynamics representation has the following beneﬁts: ﬁrst, it captures the inductive bias of the underlying physics,since real-world objects are composed of underlying atoms that can be modeled on the micro-levelby particles.
Second, we can incorporate inductive bias by directly applying the effect of the robotgripper on the particle being grasped (though the effect on the other particles must still be inferred).
Last, particle-based models are invariant to visual features such as object colors or patterns.
As such,in this paper we aim to learn a particle-based dynamics model for cloth.
However, the challengesin applying the particle-based model to cloth are that we cannot directly observe the underlying
[Figure]
particles composing the cloth nor their mesh connections.
The problem is made even more challenging due to the partial observability of the cloth from self-occlusions when it is in a crumpledconﬁguration.
Our insight into this problem is that, rather than ﬁtting a mesh model to the observation, we shouldlearn the visible connectivity dynamics (VCD): a dynamics model based on the connectivity structureof the visible portion of the cloth.
To do so, we ﬁrst learn to estimate the visible connectivity graph :we estimate which points in the point cloud observation are connected in the underlying cloth mesh.
Estimating the mesh connectivity of the observation is a simpliﬁcation of the problem of ﬁttinga single full mesh model of the entire cloth to the observation; however, it is signiﬁcantly easierto learn, since we do not need to ﬁnd a globally consistent explanation of the observation whichrequires reasoning about occlusion; to estimate the mesh connectivity of the observation, we onlyneed to consider the visible local cloth structure.
The downside of inferring the visible connectivity graph is that it cannot be used with a physics simulator during the execution of the task, since it does not capture the occluded part of the object.
Weovercome this limitation by learning a dynamics model directly on the inferred visible connectivitygraph (see Figure 1).
Unlike a physics simulator, the learned dynamics model can be trained to berobust to the partial observability of the cloth.
In this work, we focus on the task of smoothing a piece of cloth from a crumpled conﬁguration.
Wepropose a method that infers the observable particles and their connections from the point cloud,learns a visible connectivity dynamics model for the observable portion of the cloth, and uses itfor planning to smooth the cloth.
We show that for smoothing, planning with a visible connectivity dynamics model greatly outperforms state-of-the-art model-based and model-free reinforcementlearning methods that use a ﬁxed-size latent vector representation or learn a pixel-based visual dynamics model.
Furthermore, we demonstrate zero-shot sim-to-real transfer where we deploy themodel trained in simulation on a Franka arm and show that the learned model can successfullysmooth cloths of different materials, geometries, and colors from crumpled conﬁgurations.
Related Work
Vision-based Cloth Manipulation: Some papers on cloth manipulation assume that the cloth isalready lying ﬂat on the table [9, 10, 11].
If the cloth starts in an unknown conﬁguration, then oneapproach is to perform a sequence of actions that are designed to move the cloth into a set of knownconﬁgurations from which perception can be performed more easily [12, 13, 14].
For example, therobot might ﬁrst grasp the cloth by an arbitrary point and raise it into the air; it can then detect thelowest point, either while the cloth is held in the air [12, 15, 16, 17, 18, 19] or after throwing thecloth on the table [14].
By constraining the cloth to this conﬁguration set, the task of perceiving thecloth or ﬁtting a mesh model [1] is greatly simpliﬁed.
However, these funneling actions are usuallyscripted and are not generalizable to different cloth shapes or conﬁgurations.
In contrast, our workaims to enable a robot to interact with cloth from arbitrary conﬁgurations and shapes.
Other early works designed vision systems for detecting cloth features that can be used for downstream tasks, such as a Harris Corner Detector [20] or a wrinkle-detector [21].
More examples ofsuch approaches are described in [1].
However, these approaches require a task-speciﬁc manual design of vision features and are typically not robust to different variations of the cloth conﬁguration.
Policy Learning for Cloth Manipulation: Recently, there have been a number of learning basedapproaches to cloth folding and smoothing.
One approach is to learn a policy to achieve a givenmanipulation task.
Some papers approach this using learning from demonstration.
The demonstrations can be obtained using a heuristic expert [22] or a scripted sequence of actions based on clothdescriptors [23].
Another approach to policy learning is model-free reinforcement learning (RL), which has been applied to cloth manipulation [4, 3, 24].
However, policy learning approaches oftenlack the ability to generalize to novel situations; this is especially problematic for cloth manipulationin which the cloth can be in a wide variety of crumpled conﬁgurations.
We compare our method toa state-of-the-art policy learning approach [3] and show greatly improved performance.
Model-based RL for Cloth Manipulation: Model-based RL methods learn a dynamics model andthen use it for planning.
Model-based reinforcement learning methods have many beneﬁts such assample efﬁciency, interpretability, and generalizability to multiple tasks.
Previous works have triedto learn a pixel-based dynamics model that directly predict the future cloth images after an action isapplied [25, 5].
However, learning a visual model for image prediction is difﬁcult and the predictedimages are usually blurry, unable to capture the details of the cloth.
Another approach is to representthe cloth with a ﬁxed-size latent vector representation [2] and to plan in that latent space.
However,cloth has an intrinsic high dimension state representation; thus, such compressed representationstypically lose the ﬁne-grained details of their environment and are unsuitable for capturing the lowlevel details of the cloth’s shape, such as folds or wrinkles, which can be important for folding orother manipulation tasks.
Our method also falls into the model-based RL category; unlike previousworks, we learn a particle based dynamics model [6, 7], which can better capture the cloth dynamicsdue to the inductive bias of the particle representation.
Additionally, the particle representation isinvariant to visual features and enables easier sim-to-real transfer. [Figure] t = t
[Figure]
t [Figure] if t [Figure]
Method
An overview of our method, VCD (Visible Connectivity Dynamics), can be found in Figure 2.
Werepresent the cloth using a Visible Connectivity Graph, in which we infer the collision and meshedges between points of a partial point cloud.
Next, we learn a dynamics model over this graph, andﬁnally we use this dynamics model for planning robot actions.
3.1 Graph Representation of Cloth Dynamics
We represent the state of a cloth with a graph (cid:104) V, E (cid:105) .
The nodes V = { v i } i =1 ...N represent the 113 particles that compose the cloth, where v i =
( x i , ˙ x i ) denotes the particle’s current position andvelocity, respectively.
There are two types of edges E in the graph, representing two types of interactions between the particles: mesh edges and collision edges.
The mesh edges, E M , represent theconnections among the particles on the underlying cloth mesh.
The mesh connectivity is determinedby the structure of the cloth and does not change throughout time.
Each edge e ij =
( v i , v
j ) ∈
E Mconnects nodes v i to v j and models the mesh connection between them.
The other type of edges are collision edges, E C , which model the collision dynamics among two particles that are nearbyin space.
These can be different from the mesh edges due to the folded conﬁguration of the cloth,which can bring two particles close to each other even if they are not connected by a mesh edge.
Unlike the mesh edges which stay the same throughout time, these collision edges are dynamicallyconstructed at each time step based on the following criteria:
[Equation]
) where R is a distance threshold and x i,t , x j,t are the positions of particles i, j at time step t. Throughout the paper, we use the subscript t to denotes the state of a variable at time step t if the variablechanges with time.
Additionally, we assume that E M ⊂ E C , since a mesh edge connects nodes thatare close to each other and hence should also satisfy Eqn. 1.
3.2 Inferring Visible Connectivity from a Partial Point Cloud
In the real world, we observe the cloth in the form of a partial point cloud.
In this case, we representthe nodes of the graph using the partial point cloud and infer the connectivity among these observedpoints.
We denote the raw point cloud observation as P raw =
{ x i } i =1 ..N raw , where x i is the position 132 of each point and N raw is the number of points.
We ﬁrst pre-process the point cloud by ﬁlteringit with a voxel grid ﬁlter: we overlay a 3d voxel grid over the observed point cloud and then takethe centroid of the points inside each voxel to obtain a voxelized point cloud P = { x i } i =1 ,...,N p .
This preprocessing step is done both in simulation training and in the real world, which makes ourmethod agnostic to the density of the observed point cloud and more robust during sim2real transfer.
We create a graph node v i for each point x i in the voxelized point cloud P .
The collision edgesare then inferred by applying the criterion from Eqn. 1.
However, inferring the mesh edges isless straightforward, since in the real world we cannot directly perceive the underlying cloth meshconnectivity.
To overcome this challenge, we use a graph neural network (GNN) [26] to infer themesh edges from the voxelized point cloud.
Given the positions of the points in P , we ﬁrst constructa graph (cid:104) P, E C (cid:105) with only the collision edges based on Eqn. 1.
As we assume E M ⊂ E C , wethen train a classiﬁer, which is a GNN, to estimate whether each collision edge e ∈ E C is also amesh edge.
We denote this edge GNN as G edge .
The edge GNN takes as input the graph (cid:104) P, E C (cid:105) ,propagates information along the graph edges in a latent vector space, and ﬁnally decodes the latentvectors into a binary prediction for each edge e ∈ E C (predicting whether the edge is also a meshedge). For the edge GNN, we use the network architecture in previous work [7] (referred to as GNS). See Appendix A.1 for the detailed architecture.
The edge GNN is trained in simulation,where we obtain labels for the mesh edges based on the ground-truth mesh of the simulated cloth.
After training, it can then be deployed in the real world to infer the mesh edges from the point cloud.
We defer the description of how we obtain the ground-truth mesh labels in Sec. 3.5.
3.3 Modeling Visible Connectivity Dynamics with a GNN
In order to predict the effect of a robot’s action on the cloth, we must model the cloth dynamics.
While there exists various physics simulators that support simulation of cloth dynamics [27, 28,29], applying these simulators for a real cloth is still challenging due to two difﬁculties: ﬁrst, onlya partial point cloud of a crumpled cloth is observed in the real world, usually with many selfocclusions.
Second, the estimated mesh edges from Sec.
3.2 may not all be accurate.
To handlethese challenges, we learn a dynamics model based on the voxelized partial point cloud and itsinferred visible connectivity (Sec. 3.2).
Formally, given the cloth graph G t = (cid:104) V, E (cid:105) , a dynamics
GNN G dyn predicts the particle accelerations in the next time step, which can then be integratedto update the particle positions and velocities.
Here, V refers to the voxelized point cloud, and Erefers to inferred visible connectivity that includes both the predicted mesh edges E M as well as thecollision edges.
Our dynamics GNN G dyn uses the similar GNS architecture as the G edge .
It takes acloth mesh as input with state information on each node, propagates the information along the graphedges in a latent vector space, and ﬁnally decodes the latent vectors into the predicted accelerationon each node.
See Appendix A.1 for the detailed architecture of the GNN.
3.4 Planning with Pick-and-place Actions
We plan in a high-level, pick-and-place action space over the VCD model.
For each action a ={ x pick , x place } , the gripper grasps the cloth at x pick , moves to x place , and then drops the cloth.
As the GNN dynamics model is only trained to predict the changes of the particle states in small timeintervals in order to accurately model the interactions among particles, we decompose each highlevel action into a sequence of low-level movements, where each low-level movement is a small deltamovement of the gripper and can be achieved in a short time.
Speciﬁcally, we generate a sequence ofsmall delta movements ∆ x 1 , ..., ∆ x H from the high-level action, where x pick + (cid:80) Hi =1 ∆ x i = x place .
Each delta movement ∆ x i moves the gripper a small distance along the pick-and-place direction andthe motion can be predicted by the dynamics GNN in a single step.
When the gripper is graspingthe cloth, we denote the picked point as u .
We assume that the picked point is rigidly attached tothe gripper; thus, when considering the effect of the t th low-level movement of the robot gripper,we modify the graph by directly setting the picked point u ’s position x u,t
= x pick + (cid:80) ti =1 ∆ x i andvelocity ˙ x u,t = ∆ x i / ∆ t, where ∆ t is the time for one low-level movement step. The dynamics
GNN will then propagate the effect of the action along the graph when predicting future states.
Forthe initial steps where the historic velocities are not available, we pad them with zeros for input tothe dynamics GNN.
If no point is picked, e.g., after the gripper releases the picked point, then thedynamics model is rolled out without manually setting any particle state.
Our goal is to smooth a piece of cloth from a crumpled conﬁguration.
To compute the reward rbased on either the observed or the predicted point cloud, we treat each point in the point cloud as asphere with radius R and compute the covered area of these spheres when projected onto the groundplane.
Due to computational limitations, we greedily optimize this reward over the predicted statesof the point cloud after a one-step high-level pick-and-place action rather than optimizing over asequence of pick-and-place actions.
Given the current voxelized point cloud of a crumpled cloth
P , we ﬁrst estimate the mesh edges using the edge predictor E M = G edge ( (cid:104) P, E C (cid:105) ) .
We keep themesh edges ﬁxed throughout the rollout of a pick-and-place trajectory since the structure of the clothis ﬁxed.
In theory, it could be helpful to update the mesh edges based on the newly observed pointcloud at each low-level step, but this is challenging due to the heavy occlusion from the robot’s armduring the execution of a pick-and-place action.
After the execution of each pick-and-place action,new particles may be revealed and we update the mesh edges when re-planning the next action.
Thepseudocode of the planning procedure can be found in the appendix.
3.5 Training in Simulation
The simulator we use for training is Nvidia Flex, a particle-based simulator with position-baseddynamics [30, 31], wrapped in SoftGym [28].
In Flex, a cloth is modeled as a grid of particles, withspring connections between particles to model the bending and stretching constraints.
One challenge that we must address is that the points in the observed partial point cloud do not directly correspond to the underlying grid of particles in the cloth simulator.
This presents a challengefor obtaining the ground-truth labels used for training the dynamics GNN and the edge GNN, including the acceleration for each point in the observed point cloud and the mesh edges among them.
To address this issue, we perform bipartite graph matching to match each point in the voxelizedpoint cloud to a simulated particle by minimizing the Euclidean distance between the matched pairs.
Details about the matching can be found in the appendix.
After we get the mapping from the pointsto the simulator particles, the ground-truth acceleration of each point is simply assigned to be theacceleration of its mapped particle, which is used for training the dynamics GNN.
For training theedge GNN, a collision edge is assumed to be a mesh edge if the mapped simulation particles of theedge’s both end points are connected by a spring in the simulator.
3.6 Shape Speciﬁc Training
When a shape prior of the cloth is known, we further introduce two techniques to leverage suchinformation to improve the performance.
The ﬁrst is graph-based privileged imitation learning,which transfers features from a teacher model trained on the full cloth to a student model trainedon the partial cloth observation, to better allow our model to reason about occlusions.
This idea isrelated to other recent work which trains a student with partial observations to imitate a teacher withfull-state information [32, 33, 34].
Speciﬁcally, we ﬁrst train a privileged teacher dynamics modelwith ground-truth information of the particle state of the full cloth, i.e., it takes as input all particles(including the occluded particles).
Next, we train the student model, which takes a partial point cloudas input.
The student is trained to imitate the features of the corresponding nodes in the teacher [33].
Graph imitation offers direct supervision on the intermediate features and enables the student modelto implicitly reason about the occluded part of the cloth, by imitating the features from the teacher which has full state information.
This approach is shape-speciﬁc, since the occluded parts of thecloth can only be inferred if the cloth shape is known.
The second shape-speciﬁc approach that weintroduce is adding an auxiliary task of reward prediction [35], where the reward is the covered areaof the cloth after taking the current action.
Details on our shape speciﬁc training methods can befound in the appendix.
Since both methods are shape-speciﬁc, we only use them when we train andtest on the same (or similar) cloth shape; on the other hand, we remove them when we want to trainon one shape (e.g. square cloth) and test on a very different shape (e.g. T-shirt).
Experiments
4.1 Experimental Setup
Simulation Setup As mentioned, we use the Nvidia Flex simulator wrapped in SoftGym [28] fortraining.
The robot gripper is modeled as a spherical picker that can move freely in 3D space andcan be activated so the nearest particle will be attached to it.
For training, we generate random pickand-place trajectories on a square cloth.
The side length of the cloth varies from 25 to 28 cm.
Forevaluation, we consider three different shapes: 1) the same type of square cloth as used in training;2) Rectangular cloth, with its length and width sampled from [19 , 21] × [31 , 34] cm. 3)
Two layered
T-shirt (the square cloth used for training was single-layered).
For each shape, the experiment wasrun 40 times, each time with a different initial conﬁguration of the fabric. We report the 25%, 50%and 75% ( Q 25 , Q 50 , Q 75 ) percentiles of the performance.
For all our quantitative results, numbersafter ± denotes max( | Q 50 − Q 25 | , | Q 75 , − Q 50 | ) .
Our goal for cloth smoothing is to maximize the covered area of the cloth in the top-down view.
We report two performance metrics: Normalized improvement (NI) and normalized coverage(NC).
NI computes the increased covered area normalized by the maximum possible improvement
NI = s
− s 0 s max − s 0 , where s 0 , s, s max are the initial, achieved, and maximum possible covered area ofthe cloth.
Similarly, NC = s s max computes the achieved covered area normalized by the maximumpossible covered area.
We report NI in the main paper and NC in the appendix.
We evaluates two variants of our method: Visible Connectivity Dynamics (VCD) and VCD withshape-speciﬁc training.
We compare with previous state-of-the art methods for cloth smoothing:
VisuoSpatial Foresight (VSF) [5], which learns a visual dynamics model using RGBD data; Contrastive forward model (CFM) [2], which learns a latent dynamics model via contrastive learning;
Maximal Value under Placing (MVP) [3], which uses model-free reinforcement learning with aspecially designed action space.
More implementation details can be found in the appendix.
Real World Setup We use our dynamics model trained in simulation to smooth cloth in the realworld with a Franka Emika Panda robot arm and a standard panda gripper.
We obtain RGBD imagesfrom a side view Azure Kinect camera.
We use color thresholding for segmenting the cloth andobtain the cloth point cloud.
We evaluate on three pieces of cloth: Two square towels made ofcotton and silk respectively, and one t-shirt made of cotton.
We use our dynamics model trained insimulation without any ﬁne-tuning.
More details are in the appendix.
4.2 Simulation Results
For each method, we report the NI after different numbers of pick-and-place actions.
A smoothingtrajectory ends early when NI > 0.95.
We note that the edge GNN can achieve a high prediction accuracy of 0 .
91 on the validation dataset.
See appendix for visualizations of the edge GNN prediction.
We ﬁrst test all methods on the same type of square cloth used in training.
The results are shownin Figure 3 (left).
Under any given number of pick-and-place actions, VCD greatly outperforms allof the baselines.
The shape-speciﬁc training approaches described in Section 3.6 further improvesthe performance.
To test the generalization of these methods to novel cloth shapes that are not seenduring training, we further evaluate on a rectangular cloth and a t-shirt.
For this experiment weonly compare VCD to VSF, since VSF achieves the best performance on the square cloth amongall the baselines.
The results are summarized in Figure 3 (middle and right).
VCD shows a largerimprovement over VSF on the rectangular cloth.
T-shirt is more different from the training squarecloth and VSF completely fails, while VCD still shows good generalization.
The shape speciﬁctraining still leads to marginal improvement and better stability on rectangular since it has a similarshape to the square cloth.
However, as the t-shirt has very different shape compared to the squarecloth, VCD-shape-speciﬁc cannot generalize to it.
Since VCD learns a particle-based dynamics model, it incorporates the inductive bias of the clothstructure, which leads to better performance and stronger generalization across cloth shapes, compared to RGB based method like VSF.
Please see the appendix for examples of some planned pickand-place action sequences of our method on all cloth shapes as well as visualizations of the predictions of our model.
[Table]
4.3 Real-world Results
We also evaluate our method for smoothing in the real world.
We only evaluate VCD (i.e., withoutshape speciﬁc training) since it works well on all cloth shapes in simulation.
Unfortunately, we werenot able to evaluate the baselines in the real world due to the difﬁculties of transferring their RGBbased policies from simulation.
All of the baselines use RGB data as direct input to the dynamicsmodel or the learned policy, making them sensitive to the camera view and visual features.
Incontrast, our method uses a point cloud as input, which makes it robust to the camera position aswell as variation in visual features such as the cloth color or patterns.
The point cloud representationallows our method to easily transfer to the real world.
We evaluate 12 trajectories for each cloth.
The quantitative results are in Table 1 and a visualizationof smoothing sequences is shown in Figure 4.
Despite the drastic differences of the cotton and silkcloths in visual appearances, shapes, as well as the different dynamics, our model is able to smooththe cotton and silk cloths and generalize well to t-shirt.
We also report the performance if our methodis able to terminate optimally in hindsight and choose the frame with the highest performance in eachtrajectory; the result is shown in the last column of Table 1.
Videos of complete trajectories and themodel predicted rollouts can be found on our project website.
4.4 Ablation Studies
[Figure]
[Table]
We perform the following ablations to study thecontribution of each component of our method.
The ﬁrst ablation replaces the learned GNN dynamics model with the Flex simulator to testwhether a learned dynamics model performsbetter for our task than the physical simulator.
In more detail, after we use the edge GNN to infer the mesh edges on the point cloud, we createa cloth using Flex where a particle is createdat each location of the voxelized points and aspring connection is added for each inferred mesh edge.
The results is shown in Table 2, row 2.
Wesee that using the Flex simulator instead of the dynamics GNN produces signiﬁcantly worse performance.
The main reason is that the cloth created from the partial point cloud with the inferred meshedges deviates from the common cloth mesh structure used in Flex; thus, using the Flex simulatorunder this condition does not create realistic dynamics.
On the other hand, the dynamics GNN istrained directly on the partial point cloud; therefore it can learn to compensate for the partial observability when predicting the cloth dynamics.
This ablation validates the importance of using adynamics GNN to learn the dynamics of the partially observable point cloud.
The next set of ablations aims to test whether using an edge GNN to infer the mesh edges as described in Section 3.1 is necessary for learning a good dynamics model.
First, we train a dynamics
GNN without using the edge GNN, where the edges are constructed solely based on distance by
Eqn.
Since this ablation does not use an edge GNN, it cannot have two different edge types(collision vs mesh edges).
Thus at test time, all edges can either be kept ﬁxed throughout the trajectory (similar to the mesh edges in our model), or dynamically reconstructed using Eqn.
(1) at eachtime step (similar to the collision edges in our model).
The results of these two ablations are shownin Table 2, rows 3 and 4.
As can be seen, the performance is worse without the edge GNN.
Additionally, we perform another ablation where we train with both collision and mesh edges, butat test time, we do not use an edge GNN to infer the edge type; instead we consider the edges thatsatisfy the criteria of Eqn.
(1) in the ﬁrst time step as the mesh edges.
The result of this ablationis shown in Table 2, row 5.
The performance is again much worse.
All these ablations validate theimportance of using an edge GNN to infer the mesh edges.
Conclusion
In this paper, we propose the visible connectivity dynamics (VCD) model, that infers a visibilityconnectivity graph from the partial point cloud and learns a particle-based dynamics model over thegraph for planning to perform cloth smoothing.
VCD has the advantage of posing strong inductivebias that ﬁts the underlying cloth physics, being invariant to visual features, and being interpretable.
We show that VCD greatly outperforms previous state-of-the-art methods for cloth smoothing, andachieves zero-shot sim-to-real transfer on a Franka arm for smoothing various types of cloth.
Our work demonstrates the importance of the choice of state representation for efﬁcient and generalizable manipulation, as well as the beneﬁts of a graph-based representation.
While there maynot be a universal representation suitable for all objects, we believe that a graph representation canbe an important alternative to raw images or latent vectors, especially for deformable objects suchas cloth.
In the future, we hope VCD can also be applied to different types of object manipulationtasks, such as manipulation of cables, bags, and food.