  Manipulating deformable objects, such as cloth and ropes, is a long-standing
challenge in robotics: their large number of degrees of freedom (DoFs) and
complex non-linear dynamics make motion planning extremely difficult. This work
aims to learn latent Graph dynamics for DefOrmable Object Manipulation
(G-DOOM). To tackle the challenge of many DoFs and complex dynamics, G-DOOM
approximates a deformable object as a sparse set of interacting keypoints and
learns a graph neural network that captures abstractly the geometry and
interaction dynamics of the keypoints. Further, to tackle the perceptual
challenge, specifically, object self-occlusion, G-DOOM adds a recurrent neural
network to track the keypoints over time and condition their interactions on
the history. We then train the resulting recurrent graph dynamics model through
contrastive learning in a high-fidelity simulator. For manipulation planning,
G-DOOM explicitly reasons about the learned dynamics model through
model-predictive control applied at each of the keypoints. We evaluate G-DOOM
on a set of challenging cloth and rope manipulation tasks and show that G-DOOM
outperforms a state-of-the-art method. Further, although trained entirely on
simulation data, G-DOOM transfers directly to a real robot for both cloth and
rope manipulation in our experiments.
