FlingBot: The Unreasonable Effectiveness of Dynamic Manipulation for Cloth Unfolding
Abstract: High-velocity dynamic actions (e.g., ﬂing or throw) play a crucial role in our everyday interaction with deformable objects by improving our efﬁciency and effectively expanding our physical reach range.
Yet, most prior works have tackled cloth manipulation using exclusively single-arm quasi-static actions, which requires a large number of interactions for challenging initial cloth conﬁgurations and strictly limits the maximum cloth size by the robot’s reach range.
In this work, we demonstrate the effectiveness of dynamic ﬂinging actions for cloth unfolding with our proposed self-supervised learning framework, FlingBot.
Our approach learns how to unfold a piece of fabric from arbitrary initial conﬁgurations using a pick, stretch, and ﬂing primitive for a dual-arm setup from visual observations.
The ﬁnal system achieves over 80% coverage within 3 actions on novel cloths, can unfold cloths larger than the system’s reach range, and generalizes to T-shirts despite being trained on only rectangular cloths.
We also ﬁnetuned FlingBot on a real-world dual-arm robot platform, where it increased the cloth coverage over 4 times more than the quasi-static baseline did.
The simplicity of FlingBot combined with its superior performance over quasi-static baselines demonstrates the effectiveness of dynamic actions for deformable object manipulation.
Introduction
[Figure]
dynamic actions
High-velocity dynamic actions play a crucial role in our everyday interaction with deformable objects.
Making our beds in the morning is not effectively accomplished by picking up each corner of the blanket and placing them in the corresponding corners of the bed, one by one.
Instead, we are more likely to grasp the blanket with two hands, stretch it, and then unfurl it with a ﬂing over the bed.
This ﬂuid high-velocity ﬂinging action is an example of dynamic manipulation [1], which is used to improve our physical reachability and action efﬁciency – allowing us to unfold large crumpled cloths with as little as one interaction.
From goal-conditioned folding [2] to fabric smoothing [3, 4], prior works have achieved success using exclusively single-arm quasistatic interactions (e.g., pick & place) for cloth manipulation.
However, these approaches require a large number of interactions for challenging initial conﬁgurations (e.g., highly crumpled cloths) or rely on strong assumptions about the cloth (e.g., predeﬁned keypoints).
Additionally, since the robot arm cannot manipulate the cloth at locations it can’t reach, the maximum cloth size is greatly limited by the robot arm’s reach range.
In this work, we focus on the task of cloth unfolding, a typical ﬁrst step for many cloth manipulation tasks.
The goal of this task is to maximize the cloth’s coverage through interactions to expose the key visual features for downstream perception and manipulation.
Therefore, an ideal cloth unfolding approach should be:
• Efﬁcient : the approach should reach a high coverage with a small number of actions from arbitrarily crumpled initial conﬁgurations.
• Generalizable : the algorithm should not rely on heuristics (e.g., grasp predeﬁned key points, etc.).
This is especially important when unfolding is only the initial stage of a cloth perception and manipulation pipeline, where key points are not visible or severely occluded, and when the system must handle cloth types unseen during training, which may not contain the predeﬁned key points.
• Flexible Beyond the Workspace : the approach should work with cloths of different sizes, including large ones which lie outside the robot’s physical reach range.
To achieve this goal, we present FlingBot, a self-supervised algorithm that learns how to unfold cloths from arbitrary initial conﬁgurations using a pick, stretch, and ﬂing primitive for a dual-arm setup.
At each time step, the policy predicts value maps from its visual observation and picks actions greedily with respect to its value maps.
To provide the supervision signal, the system computes the difference in coverage of the cloth before and after each action – the delta-coverage – from the visual input captured by a top-down camera.
FlingBot achieves over 80% coverage within 3 actions on novel cloths and increases the cloth’s coverage by more than twice that of pick & place and pick & drag quasi-static baselines on rectangular cloths.
Our approach is ﬂexible to large cloths whose dimensions exceed the robot arm’s reach ranges and generalizes to T-shirts despite being trained on rectangular cloths.
We ﬁne-tune our approach in the real world, where, on average, it increased the cloth coverage over 4 times more than the quasi-static pick & place baseline did.
In summary:
• Our main contribution is in demonstrating the effectiveness of dynamic manipulation for cloth unfolding through our self-supervised learning framework, FlingBot.
• We propose a parameterization for the dual-arm grasp of our ﬂing primitive, which enables the application of the simple yet effective single-arm grasping technique [5, 6, 7] to dual-arm grasping while satisfying safety constraints of dual-arm systems.
The simplicity of FlingBot combined with its superior performance over quasi-static baselines further emphasize the effectiveness of dynamic actions for deformable object manipulation.
• We present a custom simulator 1 built on top of PyFlex [8, 9], a CUDA accelerated simulator, which supports the loading of arbitrarily shaped cloth meshes.
We hope this opensource simulator expands cloth manipulation research to more complex cloth types.
Related Work
A convincing argument for the addition of dynamic actions to previously exclusively quasi-static cloth manipulation pipelines would need to demonstrate their superior performance on a core cloth manipulation skill.
As a typical ﬁrst step for many cloth manipulation tasks, cloth unfolding is a popular and important problem setting for studying deformable object manipulation.
The goal of cloth unfolding is to maximize the coverage of the cloth on the workspace, which exposes key visual features of the cloth for downstream applications.
However, achieving a fully unfolded cloth conﬁguration from a severely crumpled initial conﬁguration remains a challenging problem.
Additionally, doing so efﬁciently for many different types of cloths, some larger than the system’s reach range, is extremely challenging.
Quasi-static Cloth Manipulation with Expert Demonstrations.
Prior works have explored using heuristics and identifying key points such as wrinkles [10], corners [11, 12, 13], edges [14], or combinations of them [15], but fail to generalize to severely self-occluded cloth conﬁgurations, to when required key points aren’t visible, or to non-square cloths.
Recent reinforcement learning approaches [3, 16] relied on cloth unfolding expert demonstrations in quasi-static pick-and-drag action spaces, While they sidestep the exploration problem, expert demonstrations can be sub-optimal and brittle if from hard-coded heuristics [3] or expensive if from humans [16].
Self-supervised Quasi-static Cloth Manipulation.
Bypassing the dependency on an expert, selfsupervised cloth manipulation has been demonstrated in unfolding with a factorized pick & place action space by Wu et al.
[4] and goal conditioned folding using spatial action maps [7, 6, 5] by Lee et al.
However, all these approaches operate entirely in quasi-static action spaces.
Dynamic Cloth Manipulation.
In contrast to quasi-static manipulation, which are “operations that can be analyzed using kinematics, static, and quasi-static forces (such as frictional forces at sliding contacts)”
[1], dynamic manipulation involves operations whose analysis additionally requires “forces of acceleration”.
Intuitively, dynamic manipulation (e.g., tossing [6]) involve high-velocity actions which build up objects’ momentum, such that the manipulated objects continue to move after the robot’s end-effector stops.
Such dynamic manipulation results in an effective increase in reach range and a reduction in the number of actions to complete the task compared to exclusively quasi-static manipulation.
However, prior works on dynamic cloth manipulation have either relied on cloth’s vertex positions [17] (so is only practical in simulation), a motion capturing system with markers combined with human demonstrations [18], or custom hardware [19, 20].
While prior cloth unfolding works report high average ﬁnal coverages from relative high average initial coverages, we emphasize that we consider severely crumpled initial cloth conﬁgurations, which are much more challenging yet still realistic (see section 4.2).
Additionally, no prior works have considered unfolding cloths larger than the robot’s reach range (indeed, Ganapathi et al.
[16]’s workspace is signiﬁcantly larger than their maximum cloth size due to physical reachability limitations).
Our algorithm achieves high performance in these challenging cases through self-supervised trial and error and learns directly from visual input without requiring expert demonstrations or ground truth state information.
Method
The goal of cloth unfolding is to manipulate a cloth from an arbitrarily crumpled initial state to a ﬂattened state.
Concretely, this amounts to maximizing the cloth’s coverage on the workspace surface.
Intuitively, dynamic actions have the potential to achieve high performance on cloth unfolding by appropriately making use of the cloth’s mass in a high-velocity action to unfold the cloth (Sec. 3.1).
From a top-down RGB image of the workspace with the cloth, our policy decides the next ﬂing action (Sec. 3.2) by picking the highest value action which satisﬁes the system’s constraints (Sec. 3.3).
It predicts the value of each action with a value network (Sec. 3.4) which is trained in a selfsupervised manner to take actions that maximally increases the cloth’s coverage.
The supervision signal is computed directly from the visual observation captured by the top-down camera.
After training in simulation, we ﬁnetune and evaluate the model in the real world (Sec. 3.5).
3.1 Advantages of the Fling Action Primitive
Quasi-static actions, such as pick & place, rely on friction between the cloth and ground to stretch the cloth out.
The complexity of friction forces between the workspace and the (nonvisible, groundfacing) surfaces of the cloth means the cloth’s ﬁnal conﬁguration may be difﬁcult to predict from only visual observations, especially for novel cloth types or signiﬁcantly different friction forces (i.e.: simulation v.s. real).
In addition, such systems can’t manipulate points on the cloth to regions outside of their physical reach range, which limits their maximum cloth dimensions.
Meanwhile, dynamic actions largely rely on cloths’ mass combined with a high-velocity throw to do most of its work.
Since a single dynamic motion primitive can effectively unfold many cloths, dynamic unfolding systems can learn a simpler policy which generalizes better to different cloth types.
In addition, they can reach higher coverages in smaller numbers of interactions and throw corners of cloths larger than the system’s reach range, effectively expanding their physical reach range.
We propose to use a dual-arm pick, stretch, and ﬂing primitive.
Here, a dual-arm system stretches the cloth between the arms, which unfolds it in one direction, and then ﬂings the cloth, which makes use of the cloth’s mass to unfold it in the other direction.
The combination of stretching and ﬂinging ensures that, given two appropriate grasp points, our motion primitive should be sufﬁcient for single step unfolding when possible.
3.2 Fling Action Primitive Deﬁnition
To achieve an efﬁcient, generalizable, and ﬂexible cloth unfolding system, we argue that the system requires two arms operating in a dynamic action space.
To this end, we’ve designed a pick, stretch, and ﬂing motion primitive for a dual-arm system, where each arm is placed on either side of the cloth workspace, as follows.
First, the arms perform a top-down pinch-grasp on the cloth at locations
L, R ∈ R 3 .
Second, the arms lift the cloth to 0 .
30m and stretch the cloth taut in between them.
[Figure]
Primitive
Parameterization
[Figure]
A good ﬂing could unfold a highly crumpled cloth in as little as a single step. In contrast, the quasistatic Pick & Place primitive, which grasps at the start location S , lifts, moves, then places at the end position speciﬁed by the arrow tip, requires many steps for such challenging cloth conﬁgurations. The quasi-static Pick & Drag primitive is just a Pick & Place without the lift step. Third, the arms ﬂing the cloth forward 0 .
70m at 1 .
4m s − 1 then pull backwards 0 .
50m at 1 .
4m s − 1 , which swings the cloth forwards.
Finally, the arms place the cloth down to the workspace and release their grips.
The entire motion primitive is visualized in (Fig. 2a).
Instead of parameterizing and learning all steps of our primitive, we can ﬁx the stretching step to always stretch the cloth as much as possible without tearing the cloth.
We also ﬁx the ﬂing speed and trajectory from the observation that the real world system could robustly unfold the cloth using a wide range of ﬂing parameters (i.e.: ﬂing height, ﬂing speed) if given a good grasp (see supplementary materials).
Thus, the problem of learning to pick, stretch, and ﬂing reduces down to the problem of learning where to pick, which is parameterized with only two grasp locations, one for each arm.
3.3 Constraint Satisfying Fling Action Parameterization
A dual-arm grasp is parameterized by two points L, R ∈ R 3 , which denotes where the left and right arm should approach from a top-down grasp respectively, and requires 6 scalars.
Without loss of generality for the purposes of grasping from a top-down RGB-D input, the third dimension could be speciﬁed by depth information.
This reduces L and R to two points in R 2 , each representing pixels to grasp from the visual input and uses only 4 scalars in total.
However, to minimize collisions between two arms, we wish to impose a constraint that L is always left of R , and vice versa .
Additionally, the grasp width (i.e., the length of the line L − R ) must not exceed the physical limit of the system and be smaller than the minimum safe distance limit between the two arms .
Directly using L and R will entangle these two contraints (see supplementary materials), making them difﬁcult to satisfy.
To make these constraints linear and independent, we propose an alternative 4-scalar parameterization, which consists of pixel position of the point C ∈ Rat the center of the line L − R , an angle θ ∈ R denoting the planar rotation of the line L − R , and a grasp width w ∈ R denoting the length of the line in pixel space.
To constrain L to be on the left of R , we can constrain θ ∈ [ − 90 ◦ , 90 ◦ ] , while w can be directly constrained to appropriate system limits.
3.4 Learning Delta-Coverage Maximizing Fling Actions
The naive approach for learning the action parameters (cid:104) C x , C y , θ, w (cid:105) by directly predicting thesescalars does not accommodate the best grasp’s equivariances to the cloth’s physical transformations.
However, the learner should be equipped with the inductive bias that cloths in identical conﬁgurations have identical optimal grasp points for ﬂinging relative to the cloth, regardless of the cloth’s translation, rotation, and scale.
To this end, we propose to use spatial action maps [5, 6, 7].
By predicting grasp values with constant scale and rotation in pixel space from the transformed images, spatial action maps can recover grasp values with varying scales and rotations in world space by varying the transformation applied to the image.
Concretely, given a visual observation from the top-down view (Fig. 3a), we generate a batch of rotated and scaled observations (Fig. 3b) then predict the corresponding batch of dense {+40 m / 1 . 5x 2.0x 1.5x 1.0x 90 ° 60 ° 30° 
 ° ... a) Workspace b) Rotated & Scaled Images c)+}
Predicted Value Maps d) Highest Value 12 rotations
Residual Blocks
Conv Blocks Value Network 8 s ca l e s 2.0x 1.5x 1.0x 90 ° 60 ° 30 ° 0 ° f) Fling Action e) Reachability R
L - 30 °
C ( 30°, 1.5x ) ...
From a top-down RGB image a), our policy evaluates a batch of different action rotations and scales by transforming the observation b) then predicting the corresponding batch of value maps c).
The highest value action d), corresponding to the maximum value pixel which also satisﬁes the arms’ reachability constraints e) is chosen.
Finally, the chosen pixel’s location and its observation’s transformation is decoded into the ﬂing action parameters (i.e., center point, distance, relative orientation between gripper).
value maps (Fig. 3c).
Each pixel in each value map contains the value of the action parameterized by that pixel’s location, giving C , and its observation’s rotation and scaling, giving θ and w respectively (Fig. 3f).
The value network is supervised to regress each pixel in the value map to the ground truth delta-coverage – the difference in coverage before and after the action.
Thus, by picking the grasping action with the highest value (Fig. 3d), the system picks grasp points which it expects would lead to the greatest increase in cloth coverage.
By its architecture, the value network is invariant to translation.
By rotating and scaling its inputs, it is also invariant to rotation and scale.
In our approach, we use 12 rotations, which discretizes the range [ − 90 ◦ , 90 ◦ ] , and 8 scale factors in the range [1 . 00 , 2 . 75] at 0 .
25 intervals.
Our value network is a fully convolutional neural network with nine residual blocks [21] and two convolutional layers in the ﬁrst and last layer, and takes as input 64 × 64 RGB images.
To evaluate actions within a useful range of scales, we crop the image such that the cloth takes up at most two-thirds of the image width and height before shrinking it by the inverse of the 8 scale factors above.
We also ﬁlter out and reject all grasp point pairs which are out of reach for either arm.
Self-Supervised Learning.
The value network is trained end-to-end with self-supervised trials in simulation then ﬁnetuned in the real world.
In each training episode, a random task is sampled.
Each ﬂing step is labeled with its normalized delta coverage, which is computed by counting cloth mask pixels from a top-down view, then dividing by the cloth mask pixel count of a ﬂattened cloth state.
While our experiments have plain black workspace surface colors and non-textured colored cloths for easy masking, our approach can be combined with image segmentation or background subtraction approach to obtain the corresponding cloth mask to work with arbitrary cloth and workspace textures.
Note that this supervision signal (i.e., cloth mask) is only needed during training.
The policy interacts with the cloth until it reaches 10 timesteps or predicts grasps on the workspace.
The latter stopping condition indicates that the policy does not expect any possible action to further improve the cloth coverage.
All policies are trained in simulation until convergence, which takes around 150,000 simulation steps, or 6 days on a machine with a GTX 1080 Ti.
The network is trained using the Adam optimizer with a learning rate of 1e-3 and a weight decay of 1e-6.
3.5 Experiment Setup
Simulation Setup.
Our custom simulation environment is built on top of the PyFleX [8] bindings to Nvidia FleX provided by SoftGym [9].
On top of the functionality provided by PyFleX and SoftGym, our custom simulator 2 can load arbitrary cloth meshes, such as T-shirts, through the Python API.
While our simulation environment does not support loading URDFs of robots, we found it sufﬁcient to represent only the arms’ end effectors, and apply appropriate physical constraints to their locations.
The observations are rendered using Blender 3D where the cloth HSV color is sampled uniformly between [0 . 0 , 1 . 0] , [0 . 0 , 1 . 0] , and [0 . 5 , 1 . 0] respectively, and the background is dark grey with a procedurally generated normal map to mimic the wrinkles on the real-world workspace surface.
We further apply brightness, contrast, and hue jittering on observations to help with the transfer to real.
We manually tuned our simulation ﬂing speed to qualitatively match the real-world cloth dynamics (see supplementary materials on simulation sensitivity and real world robustness to ﬂing parameters).
Real World Setup.
Our real-world experiment setup consists of two UR5s, where one is equipped with a Schunk WSG50 and the other with an OnRobot RG2, facing each other and positioned 1 .
35m apart.
The top-down RGB-D image is captured with an Intel RealSense D415.
To help with pinch grasp success in real, we used a 2-inch rubber mat as the workspace and added sponge tips to the gripper ﬁngers.
To mitigate the reliance on specialized and expensive force sensor, we chose to implement stretched cloth detection using only a second Intel RealSense D415 capturing a frontal view of the workspace.
By segmenting the cloth from the front camera RGBD view of the workspace, then checking whether the top edge of the cloth mask is ﬂat as a proxy for the cloth being stretched, we can pull the arms’ end effectors apart until the top of the cloth is no longer bent.
Evaluation
We design a series of experiments to evaluate the advantage of dynamic actions over quasi-static actions in the task of cloth unfolding.
The system’s performance is evaluated on its efﬁciency (i.e., reaching high coverages with a small number of actions), generalization to unseen cloth types (i.e., ﬂattening different types of T-shirts when only trained on rectangle cloths), and reach range (i.e., on cloths with dimensions larger than its reach range).
Finally, we evaluate the algorithm’s performance on the real-world setup.
Please visit https://ﬂingbot.cs.columbia.edu for experiment videos.
4.1 Metrics
The algorithm performance is measured by the ﬁnal coverage, delta-coverage from initial coverage, and the number of interactions.
Final coverage measures the coverage at the end of an episode, whereas delta-coverage is ﬁnal coverage minus initial coverage.
All our coverage statistics are normalized (i.e., divided by the maximum possible coverage of the cloth in a ﬂattened conﬁguration) and can be easily computed from a top-down camera, while the number of interactions is the episode length.
To evaluate our policy, we load a task from the testing task datasets then run the policy for 10 steps or until the policy predicts grasps on the ﬂoor.
4.2 Task Dataset Generation
Each task is speciﬁed by a cloth mesh, mass, stiffness, and initial conﬁguration.
The cloth mesh is sampled from one of three types:
Normal Rect , which contains rectangular cloths with size within the reach range.
Edge lengths are sampled from [0 .
40m , 0 .
65m] .
Large Rect , which contains rectangular cloths with at least one edge larger than the reach range ( 0 .
70m ). Otherwise, edge sizes are sampled from [0 .
40m , 0 .
75m] , which means the shorter edge could still be smaller than the reach range.
Shirt , which contains a subset of shirts sampled from CLOTH3D’s [22] test split, all of which are resized to be within the reach range.
These shirts include tank tops, crop tops, short and long sleeves.
The cloth mass is sampled from [0 . 2kg , 2 . 0kg] and an internal stiffness from [0 . 85kg / s 2 , 0 .
95kg / s 2 ] .
Finally, the cloth’s initial conﬁguration is varied by holding a randomly grasped the cloth at a random height between [0 . 5m , 1 . 5m] then dropping and allowing the cloth to settle (similar to Lee et al.
and tier-3 in Seita et al.) , resulting in a severely crumpled conﬁguration.
To calculate the normalized coverage, we use the maximum possible coverage of the cloths in their ﬂattened conﬁgurations.
For rectangular cloths in simulation, the ﬂattened conﬁguration can be analytically calculated using the undeformed vertex positions of the grid mesh, which means the normalized coverage could still be higher if the cloth rests in a stretched position due to friction.
For shirts in simulation, we opted to calculate its maximum possible coverage as its outer surface area divided by 2, since a qualitatively ﬂattened T-shirt may not actually maximize coverage.
While this choice also makes it possible for the normalized coverage to be greater than 1, it will still preserve performance rankings.
For real world experiments, the ﬂattened conﬁgurations are manually set.
We emphasize the difﬁculty of our unfolding tasks, where cloths in Normal Rect, Large Rect, and Shirt have average initial coverages of 28.8%, 27.1%, and 46.4% in simulation respectively, and 26.1%, 33.6%, and 24.0% in the real world respectively.
In contrast, simulated cloths from Seita et al. start at 77 .
6% , and 42 .
0% for easy, medium, and hard tasks, while real world cloths
F li ng
Init
N o r m a l R ec t
( R e a l )
P &
P
F li ng
L a r g e R ec t ( R e a l )
P &
P
F li ng
Sh i r t ( R e a l )
P &
P
Step
Step0.
Init
Step
Step0.
Done 0.
Done
Done 0. Done 0.
Step
Step0.
from Ganapathi et al. starts at 71.4% and 68.4% on two different real-world trials. Yet, the challenging cloth conﬁgurations found in our tasks (Fig. 4) are realistic and prevalent in typical households. In simulation, the policy is trained on 2000 rectangular cloths sampled evenly between Normal Rect and Large Rect, and evaluated on 600 novel tasks split evenly between Normal Rect, Large Rect, and Shirt cloths.
In real, the simulation policy is deployed to collect real world experience on Normal Rect episodes (257 steps), optimized on both simulation and real world data, then evaluated on 10 novel test tasks in each cloth type. 4.3 Approach Comparisons • Quasi-static : [Pick & Place] and [Pick & Drag] are policies which uses a quasi-static “pick and place” (similar to Lee et al.
[2], visualized in Fig. 2b) and “pick and drag” (similar to Seita et al. , with no lift step compared to pick and place) primitive respectively. [Stretch & Drag] is identical to [Pick & Drag] with an extra stretch step (identical to FlingBot’s stretch) after picking.
These baselines are implemented with dual-arm set up, therefore provides the same physical reach range as our dual-arm ﬂing system.
• Dynamic manipulation : [FlingBot] is our policy which predicts the optimal two-arm grasp locations for a ﬁxed stretching and ﬂinging routine described in Sec. 3.2.
• Dynamic manipulation with ﬂing speed prediction : [FlingBot-S] is identical to [FlingBot] with an additional ﬂing speed module, which predicts the ﬂing speed within the range of [0 .
1m s − 1 , 1 . {+
+}
m s − 1 ] from visual input of the cloth after stretching and lifting.
This ﬂing speed module is trained using Deep Deterministic Policy Gradients (DDPG) [24] on the delta-coverage rewards to maximize single-step returns (discount factor γ is set to 0), where the grasps are predicted by a converged and frozen [FlingBot] value network.
However, despite the cloth physical parameter variations in the training and testing dataset (Sec. 4.2),
Tab. 1 and Fig.
5 suggest that there aren’t signiﬁcant performance gains for [FlingBot-S]
[Figure] [Figure] [Figure]
Sim
[Figure]
within a few interaction steps, while the quasi-static baselines never reach high coverages even with signiﬁcantly more interaction steps. This demonstrates the difﬁculty of unfolding from highly crumpled initial conﬁgurations and dynamic action’s superior efﬁciency.
over [FlingBot] within the cloth parameter ranges tested (Sec. 4.2).
These results justify [FlingBot]’s usage of a single ﬂing speed, even for varying cloth mass, stiffness, and size.
Thus, we prefer the simpler [FlingBot] approach for comparisons with baselines.
• Dynamic manipulation with ﬂing parameter regression : [Fling-Reg] is identical to [FlingBot], but directly predicts the action parameters (cid:104) C x , C y , θ, w (cid:105) from visual inputs instead of exploiting the task’s equivariances.
Its policy is trained using DDPG on deltacoverage rewards to maximize single-step returns.
However, from Tab. 1, [Fling-Reg] completely fails to perform the task, demonstrating the advantage of encoding inductive biases which leverage equivariances in the problem structure
4.4 Results
Better Efﬁciency.
In this experiment, we compare the unfolding efﬁciency between quasi-static and dynamic actions.
From the Normal Rect column in Tab. 1, [FlingBot] increases the coverage of the cloth ( +63 . 1% ) by more than two times that of [Pick&Place] and [Pick&Drag] baselines ( +29 .
2% , and +24 .
Additionally, from Fig. 5, [FlingBot] achieves over 80% within 3 interactions (simulation normal cloth), while the quasi-static baselines never reach such a high coverage even with signiﬁcantly more interaction steps or with a stretching subroutine.
This demonstrates the difﬁculty of unfolding from highly crumpled initial conﬁgurations and dynamic action’s superior efﬁciency. [Table]
Increased Reach Range.
In this experiment, we investigate these approaches’ performance on cloths which have dimensions larger than the robot arm’s reach range.
These tasks are not only challenging for quasi-static baselines, because the arms can’t manipulate the cloth at locations beyond its reach range, but also for our ﬂinging policies, because the arms can’t fully stretch or lift the cloth off the ground.
Despite these challenges, [FlingBot] achieves 79 .
2% (Tab. 1, column Large Rect).
Compared to the quasi-static baselines, [FlingBot] increases the coverage by +52 .
0% , which is roughly twice that of the quasi-static baselines ( +27 .
1% , +24 .
8% , +23 . 1% ).
These results show how high-velocity actions could effectively expand the physical reach range, allowing the system to be more ﬂexible with extreme cloth sizes.
Generalize to Unseen Cloth Types.
In this experiment, we investigate how well these approaches, trained on only rectangular cloths, can generalize to unseen cloth types (i.e.: T-shirts).
Qualitative real world (Fig. 4) and simulation (in supplementary materials) results suggest that our ﬂinging policy has learned to grasp keypoints on cloth (i.e., corners, edges) when it sees them, or otherwise ﬂing to reveal these features.
FlingBot’s generalization performance (93.3% in Tab. 1, column Shirt) to novel cloth geometries can be attributed to this strategy, since a cloth of any type can be unfolded by grabbing one of its edges, stretching, then ﬂinging it in a direction perpendicular to the edge.
Through self-supervised exploration, our approach discovered grasping strategies which were manually designed in heuristic based prior works, while being more generalizable to different cloth conﬁgurations and types.
Meanwhile, all quasi-static baselines exhibited worse cloth unfolding efﬁciency.
The Sim Shirts plot in Fig.
5 shows that our ﬂinging policies take only 3 actions to reach their maximum coverages, while the quasi-static baselines take upwards of 8 steps to reach lower maximum coverages.
[Table]
Evaluating Real-World Unfolding.
Finally, we ﬁnetune and evaluate our simulation models from Tab.with real-world experience on a pair of UR5 arms.
Task generation is automated using the robot arms by randomly grasping the cloth at height 0 . 50m then dropping it back on the workspace.
We use a 0 .
35m × 0 .
45m cloth for Normal Rect, a 0 .
40m × 0 .
70m bath towel for Large Rect, and a 0 .
45m × 0 .
55m T-shirt for Shirt.
The system collected 257 experience steps over 150 cloth tasks for ﬁnetuning in total.
The performance is reported averaged overtest episodes, where real-world grasp errors are ﬁltered out (see “Real World Failure Cases” below).
From Tab. 2, we report that our policy achieves over 80% on all cloth types, which outperforms the quasi-static pick & place baseline by over 40% .
Additionally, we report that the pre-ﬁnetune performance of FlingBot on Normal Cloths is 69 . 8% , justifying our decision to ﬁnetune to get a 12 . 1% improvement.
Overall, our ﬂinging primitive takes a median time of 16 .
7s .
While the pick & place primitive only takes a median time of 8 . 8s , it is unable to reach the high coverages even with many more interaction steps (Fig. 5, bottom row).
Both primitives incur additional overheads of 2 .
5s for preparing the transformed observation batch and 2 .
9s for a routine which checks whether the cloth is stuck to the gripper after the gripper is opened (details in supplementary materials).
Real World Failure Cases.
Grasping failures, where the policy speciﬁed a grasp point on the cloth but the grippers failed to successful pinch grasp, constituted all of our real-world pipeline failure cases. Our real world system uses its frontal RGBD view to detect grasp failures after the dual arm lifts the cloths up and automatically discards these episodes. The average grasp success rate is 78.0%, 45.0%, and 75.8% for normal rectangular, large rectangular, and shirts respectively. We used a bath towel for our large rectangular cloths, which is signiﬁcantly thicker and stiffer than the tea towel and T-shirt we used for normal rect and shirts.
Therefore, pinch grasps with large cloths failed signiﬁcantly more.
Additionally, even on successful grasps, the gripper may hold the cloth tight in its crumpled state, rendering all ﬂings ineffective.
However, we do not discard of these episodes.
We discuss more of real world grasp failures and show simulation failure cases in the supplementary materials .
Conclusion and Future Work
We proposed a dynamic ﬂing motion primitive and a self-supervised learning algorithm for learning the grasp parameters for the cloth unfolding task.
The cloth unfolding policy is efﬁcient, generalizable, and works with cloth sizes beyond the reach range of the system in both simulation and the real world.
While we’ve demonstrated the effectiveness of dynamic actions for cloth unfolding, dynamic actions alone are insufﬁcient for more complex deformable object manipulation tasks, such as goalconditioned folding.
In this direction, future work could explore learning in combined action spaces (i.e., dynamic and quasi-static, two-arm and single-arm, etc.) and integrating it with vision-based cloth pose estimation [25] for goal-conditioned cloth manipulation. Acknowledgments
The authors would like to thank Eric Cousineau, Benjamin Burchﬁel Naveen Kuppuswamy, other researchers in Toyota Research Institute, Zhenjia Xu, and Cheng Chi for their helpful feedback and fruitful discussions. We would also like to thank Google for their donation of UR5 robot hardware. This work was supported in part by the Amazon Research Award and the National Science Foundation under CMMI-2037101.