FlingBot: The Unreasonable Effectiveness of Dynamic Manipulation for Cloth Unfolding
Anonymous Author(s)
AffiliationAddress email
Abstract: High-velocity dynamic actions (e.g., fling or throw) play a crucial role in oureveryday interaction with deformable objects by improving our efficiency and effectivelyexpanding our physical reach range.
Yet, most prior works have tackled cloth manipulationusing exclusively single-arm quasi-static actions, which requires a large number of interactions for challenging initial cloth configurations and strictly limits the maximum clothsize by the robot’s reach range.
In this work, we demonstrate the effectiveness of dynamicflinging actions for cloth unfolding with our proposed self-supervised learning framework,
FlingBot.
Our approach learns how to unfold a piece of fabric from arbitrary initialconfigurations using a pick, stretch, and fling primitive for a dual-arm setup from visual observations.
The final system achieves over 80% coverage within 3 actions on novel cloths,can unfold cloths larger than the system’s reach range, and generalizes to T-shirts despite being trained on only rectangular cloths.
We also finetuned FlingBot on a real-world dual-armrobot platform, where it increased the cloth coverage over 4 times more than the quasi-staticbaseline did.
The simplicity of FlingBot combined with its superior performance overquasi-static baselines demonstrates the effectiveness of dynamic actions for deformableobject manipulation.
Code, data, and simulation environment will be publicly available.
Introduction
[Figure]
our beds in the morning is not effectivelyaccomplished by picking up each corner of the
ket and placing them in the cornersof the bed, one by one.
Instead, we are more likelyto grasp the blanket with two hands, stretch it, andthen unfurl it with a fling over the bed.
This fluidhigh-velocity flinging action is an example of dynamic manipulation [1], which is used to improveour physical reachability and action efficiency – allowing us to unfold large crumpled cloths with aslittle as one interaction.
From goal-conditioned folding [2] to fabricsmoothing [3, 4], prior works have achieved successusing exclusively single-arm quasi-static interactions(e.g., pick & place) for cloth manipulation.
However,these approaches require a large number of interactions for challenging initial configurations (e.g.,highly crumpled cloths) or rely on strong assumptions about the cloth (e.g., predefined keypoints).
Additionally, since the robot arm cannot manipulate thecloth at locations it can’t reach, the maximum cloth size is greatly limited by the robot arm’s reach range.
In this work, we focus on the task of cloth unfolding, a typical first step for many cloth manipulation tasks.
The goal of this task is to maximize the cloth’s coverage through interactions to expose the key visual featuresfor downstream perception and manipulation.
Therefore, an ideal cloth unfolding approach should be:
• Efficient : the approach should reach a high coverage with a small number of actions from arbitrarily
crumpled initial configurations.
[Figure]
• Generalizable : the algorithm should not rely on heuristics (e.g., grasp predefined key points, etc.).
This
is especially important when unfolding is only the initial stage of a cloth perception and manipulation
pipeline, where key points are not visible or severely occluded, and when the system must handle clothtypes unseen during training, which may not contain the predefined key points.
Flexible Beyond the Workspace : the approach should work with cloths of different sizes, including largeones which fall outside the robot’s physical reach range.
To achieve this goal, we present FlingBot, a self-supervised algorithm that learns how to unfold a piece offabric from arbitrary initial configurations using a pick, stretch, and fling primitive for a dual-arm setup.
Ateach time step, the policy predicts value maps from its visual observation and picks actions greedily withrespect to its value maps.
To provide the supervision signal, the system computes the difference in coverage ofthe cloth before and after each action – the delta-coverage – from the visual input captured by a top-downcamera.
FlingBot achieves over 80% coverage within 3 actions on novel cloths and increases the cloth’scoverage by more than twice that of pick & place and pick & drag quasi-static baselines on rectangular cloths.
Our approach is flexible to large cloths whose dimensions exceed the robot arm’s reach ranges and generalizesto T-shirts despite being trained on
rectangular cloths
We fine-
tune
our approach in the real world, where, onaverage
, it
increased
the cloth coverage over
4 times more than the quasi-static pick & place baseline did.
In summary, our contributions are three-fold:
• Our main contribution is in demonstrating the effectiveness of dynamic manipulation for cloth unfoldingthrough our self-supervised learning framework,
FlingBot.• We propose a parameterization for the dual-arm grasp of our fling primitive, which enables the applicationof the simple yet effective single-arm grasping technique spatial action maps [5, 6, 2, 7] to dual-armgrasping while satisfying safety constraints of dual-arm systems.
The simplicity of FlingBot combined withits superior performance over quasi-static baselines further emphasize the effectiveness of dynamic actionsfor deformable object manipulation.
69 • We present a custom simulator built on top of PyFlex [8, 9], a particle-based CUDA accelerated simulator,which supports the loading of arbitrarily shaped cloth meshes.
We hope this open-source simulator expands
cloth manipulation research to more complex cloth types.
Related Work
A convincing argument for the addition of dynamic actions to previously exclusively quasi-static clothmanipulation pipelines would need to demonstrate their superior performance on a core cloth manipulationskill.
As a typical first step for many cloth manipulation tasks, cloth unfolding is a popular and importantproblem setting for studying deformable object manipulation.
The goal of cloth unfolding is to maximizethe coverage of the cloth on the workspace, which exposes key visual features of the cloth for downstreamapplications.
However, achieving a fully unfolded cloth configuration from a severely crumpled initialconfiguration remains a challenging problem.
Additionally, doing so efficiently for many different types ofcloths, some larger than the system’s reach range, is extremely challenging.
Quasi-static Cloth Manipulation with Expert Demonstrations.
Prior works have explored using heuristics and identifying key points such as wrinkles [10], corners [11, 12, 13], edges [14], or combinations ofthem [15, 15], but fail to generalize to severely self-occluded cloth configurations, to when required key points aren’t visible, or to non-square cloths.
Recent works which use reinforcement learning have relied onexpert demonstrations to learn cloth unfolding [3, 16] in quasi-static pick-and-drag action spaces.
Althoughsuch approaches bypass the sample-inefficiency of the exploration problem, expert demonstrations can besub-optimal and brittle if from hard-coded heuristics [3] or expensive if from humans [16].
Self-supervised Quasi-static Cloth Manipulation.
Bypassing the dependency on an expert, self-supervisedcloth manipulation has been demonstrated in unfolding with a factorized pick & place action space by Wuet al.
[4] and goal conditioned folding using spatial action maps [7, 6, 5] by Lee et al.
However, all theseapproaches operate entirely in quasi-static action spaces.
Dynamic Cloth Manipulation.
In contrast to quasi-static manipulation, which are “operations that can beanalyzed using kinematics, static, and quasi-static forces (such as frictional forces at sliding contacts)”
[1],dynamic manipulation involves operations whose analysis additionally requires “forces of acceleration”.
Intuitively, dynamic manipulation (e.g., tossing [6]) involve high-velocity actions which build up objects’momentum, such that the manipulated objects continue to move after the robot’s end-effector stops.
Suchdynamic manipulation results in an effective increase in reach range and a reduction in the number of actions tocomplete the task compared to exclusively quasi-static manipulation.
However, prior works on dynamic clothmanipulation have either relied on cloth’s vertex positions [17] (so is only practical in simulation), a motioncapturing system with markers combined with human demonstrations [18], or custom hardware [19, 20].
While prior cloth unfolding works report high average final coverages from relative high average initialcoverages, we emphasize that we consider severely crumpled initial cloth configurations, which are muchmore challenging yet still realistic (see section 4.2).
Additionally, no prior works have considered unfoldingcloths larger than the robot’s reach range (indeed, Ganapathi et al.
[16]’s workspace is significantly largerthan their maximum cloth size due to physical reachability limitations).
Finally, our algorithm learns throughself-supervised trial and error directly from visual input without requiring expert demonstrations or groundtruth state information.
Method
The goal of cloth unfolding is to manipulate a cloth from an arbitrarily crumpled initial state to a flattenedstate.
Concretely, this amounts to maximizing the coverage of the cloth on the workspace surface.
Intuitively,dynamic actions have the potential to achieve high performance on cloth unfolding by appropriately makinguse of the cloth’s mass in a high-velocity action to unfold the cloth (Sec. 3.1).
From a top-down RGB image of the workspace with the cloth, our policy decides the next fling action(Sec. 3.2) by picking the highest value action which satisfies the system’s constraints (Sec. 3.3).
It predicts thevalue of each action with a value network (Sec. 3.4) which is trained in a self-supervised manner to takeactions that maximally increases the cloth’s coverage.
The supervision signal is computed directly from thevisual observation captured by the top-down camera.
After training in simulation, we finetune and evaluatethe model in the real world (Sec. 3.5).
3.1 Advantages of the Fling Action Primitive
Quasi-static actions, such as pick & place, rely on strategically chosen pick and place points and frictionbetween the cloth and ground to stretch the cloth out.
The complexity of friction forces between theworkspace and the (nonvisible, ground-facing) surfaces of the cloth when being dragged means the cloth’sfinal configuration may be difficult to predict from only visual observations, which may be reflected in thecomplexity of quasi-static unfolding policies.
In addition, such systems can’t manipulate points on the cloth toregions outside of their physical reach range, which limits their maximum cloth dimensions.
Meanwhile,dynamic actions largely rely on cloths’ mass combined with a high-velocity throw to do most of its work.
Specifically, a single dynamic motion primitive can be effectively reused for most instances, so such systemscan afford to learn a simpler grasping policy which generalizes better to different cloth types.
In addition, theycan reach higher coverages in smaller numbers of interactions and throw corners of very large cloths beyondthe system’s reach range, effectively expanding the reach range of the system.
We propose to use a dual-arm pick, stretch, and fling primitive.
Here, a dual-arm system stretches the clothbetween the arms, which unfolds it in one direction, and then flings the cloth, which makes use of the cloth’smass to unfold it in the other direction.
The combination of stretching and flinging ensures that, given twoappropriate grasp points, our motion primitive should be sufficient for single step unfolding when possible.
3.2 Fling Action Primitive Definition
To achieve an efficient, generalizable, and flexible cloth unfolding system, we argue that the system requirestwo arms operating in a dynamic action space.
To this end, we’ve designed a pick, stretch, and fling motionprimitive for a dual-arm system, where each arm is placed on either side of the cloth workspace, as follows.
40 m / 1 .
[Figure] [Figure]
1.5x 1.0x 90 ° 60 ° 30° 0 ° ... a) Workspace b) Rotated & Scaled Images c)
Predicted Value Maps d) Highest Value 12 rotations
Residual Blocks
Conv Blocks
Value Network 8 s ca l e s 2.0x 1.5x 1.0x 90 ° 60 ° 30 ° 0 ° f)
Fling Action e) Reachability
R
L - 30 °
C ( 30°, 1.5x ) ...
First, the arms perform a top-down pinch-grasp on the cloth at locations L , R ∈ R 3 .
Second, the arms lift thecloth to 0 .
30m and stretch the cloth taut in between them.
Third, the arms fling the cloth forward 0 .
70m at1 .
2ms − 1 then pull backwards 0 .
50m at 1 .
2ms − 1 , which swings the cloth forwards.
Finally, the arms placethe cloth down to the workspace and release their grips.
The entire motion primitive is visualized in (Fig. 2a).
Instead of parameterizing and learning all steps of our primitive, we can fix the stretching step to alwaysstretch the cloth as much as possible without tearing the cloth.
Further, a fixed flinging trajectory with afixed flinging velocity can still be effective for typical cloth masses, sizes, and stiffnesses in our task datasets(see
Sec. 4.2).
To verify this claim, we also implemented a variant of FlingBot which also predicts the flingvelocity (see Sec. 4.3).
Thus, the problem of learning to pick, stretch, and fling reduces down to the problemof learning where to pick, so we parameterize our flinging primitive with only two grasp locations, one foreach arm in the dual-arm system.
3.3 Constraint Satisfying Fling Action Parameterization
A dual-arm grasp is parameterized by two points L , R ∈ R 3 , which denotes where the left and right armshould approach from a top-down grasp respectively, and requires 6 scalars.
Without loss of generality forthe purposes of grasping from a top-down RGB-D input, the third dimension could be specified by depthinformation.
This reduces L and R to two points in R 2 , each representing pixels to grasp from the visual inputand uses only 4 scalars in total.
However, to minimize collisions between two arms, we wish to impose a constraint that L is always left of
R , and vice versa.
Additionally, the grasp width (i.e., the length of the line L − R ) must not exceed the physicallimit of the system and be smaller than the minimum safe distance limit between the two arms.
Directly using
L and R will entangle these two contraints (visualized in the supplementary material), making them difficult tosatisfy.
To make these constraints linear and independent, we propose an alternative 4-scalar parameterization,which consists of pixel position of the point C ∈ R 2 at the center of the line L − R , an angle θ ∈ R denotingthe planar rotation of the line L − R , and a grasp width w ∈ R denoting the length of the line in pixel space.
Toconstrain L to be on the left of R , we can constrain θ ∈ [ − 90 ◦ , 90 ◦ ] , while w can be directly constrained toappropriate system limits.
3.4 Learning Delta-Coverage Maximizing Fling Actions
The naive approach for learning the action parameters (cid:104) C x , C y , θ , w (cid:105) by directly predicting these 4 scalars doesnot accommodate the best grasp’s equivariances to the cloth’s physical transformations.
However, the learnershould be equipped with the inductive bias that cloths in identical configurations have identical optimal grasppoints for flinging relative to the cloth, regardless of the cloth’s translation, rotation, and scale.
To this end, we propose to use spatial action maps [5, 6, 7].
By predicting grasp values with constant scaleand rotation in pixel space from the transformed images, spatial action maps can recover grasp values withvarying scales and rotations in world space by varying the transformation applied to the image.
Concretely,given a visual observation from the top-down view (Fig. 3a), we generate a batch of rotated and scaledobservations (Fig. 3b) then predict the corresponding batch of dense value maps (Fig. 3c).
Each pixel ineach value map contains the value of the action parameterized by that pixel’s location, giving C , and itsobservation’s rotation and scaling, giving θ and w respectively (Fig. 3f).
The value network is supervised toregress each pixel in the value map to the ground truth delta-coverage – the difference in coverage before andafter the action.
Thus, by picking the grasping action with the highest value (Fig. 3d), the system picks grasppoints which it expects would lead to the greatest increase in cloth coverage.
By its architecture, the valuenetwork is invariant to translation.
By rotating and scaling its inputs, it is also invariant to rotation and scale.
In our approach, we use 12 rotations, which discretizes the range [ − 90 ◦ , 90 ◦ ] , and 8 scale factors in therange [ 1 . 00 , 2 . 75 ] at 0 .
25 intervals.
Our value network is a fully convolutional neural network with nineresidual blocks [21] and two convolutional layers in the first and last layer, and takes as input 64 × 64 RGBimages.
To evaluate actions within a useful range of scales, we crop the image such that the cloth takes up atmost two-thirds of the image width and height before shrinking it by the inverse of the 8 scale factors above.
We also filter out and reject all grasp point pairs which are out of reach for either arm.
Self-Supervised Learning.
The value network is trained end-to-end with self-supervised trials in simulationthen finetuned in the real world.
In each training episode, a random task is sampled.
Each fling step is labeledwith its unfolding performance (i.e., delta coverage).
The delta coverage is normalized, and computed bycounting cloth mask pixels from a top-down view, then dividing by the cloth mask pixel count of a flattenedcloth state.
While our experiments have plain black workspace surface colors and non-textured coloredcloths for easy masking, our approach can be combined with image segmentation or background subtractionapproach to obtain the corresponding cloth mask to work with arbitrary cloth and workspace textures.
Notethat this supervision signal (i.e., cloth mask) is only needed during training.
The policy interacts with the cloth until it reaches 10 timesteps or predicts grasps on the workspace.
Thelatter stopping condition indicates that the policy does not expect any possible action to further improve thecloth coverage.
All policies are trained in simulation until convergence, which takes around 90,000 simulationsteps, or 4 days on a machine with a GTX 1080 Ti.
The network is trained using the Adam optimizer with alearning rate of 1e-3 and a weight decay of 1e-6.
3.5 Experiment Setup
Simulation Setup.
Our custom simulation environment is built on top of the PyFleX [8] bindings to Nvidia
FleX provided by SoftGym [9].
On top of the functionality provided by PyFleX and SoftGym, our simulationcan load arbitrary cloth meshes, such as T-shirts, through the Python API and will be released along withcode and data.
While our simulation environment does not support loading URDFs of robots, we found itsufficient to represent only the arms’ end effectors, and apply appropriate physical constraints to their locations.
Specifically, we precompute a reachability map for each arm using PyBullet [22], and check each proposedgrasp location against the corresponding reachability map.
The observations are rendered using Blender3D where the cloth HSV color is sampled uniformly between [ 0 . 0 , 1 . 0 ] , [ 0 . 0 , 1 . 0 ] , and [ 0 . 5 , 1 . 0 ] respectively,and the background is dark grey with a procedurally generated normal map to mimic the wrinkles on thereal-world workspace surface.
We further apply brightness, contrast, and hue jittering on observations in thedata loader to help with the transfer to real.
To qualitatively match the real-world cloth dynamics, our flingingprimitive in simulation flings with speed 0 .
60ms − 1 instead of the real world fling speed 1 .
20ms − 1 .
Real World Setup.
Our real-world experiment setup consists of two UR5s, where one is equipped with a
Schunk WSG50 and the other with an OnRobot RG2, facing each other and positioned 1 .
35m apart.
Thetop-down RGB-D image is captured with an Intel RealSense D415.
To help with pinch grasp success in real,we used a 2-inch rubber mat as the workspace and added sponge tips to the gripper fingers.
To mitigate thereliance on specialized and expensive force sensor, we chose to implement stretched cloth detection using onlya second Intel RealSense D415 capturing a frontal view of the workspace.
By segmenting the cloth from thefront camera RGBD view of the workspace, then checking whether the top edge of the cloth mask is flat as aproxy for the cloth being stretched, we can pull the arms’ end effectors apart until the top of the cloth is nolonger bent.
Evaluation
We design a series of experiments to evaluate the advantage of dynamic actions over quasi-static actions in thetask of cloth unfolding.
The system’s performance is evaluated on its efficiency (i.e., reaching high coverageswith a small number of actions), generalization to unseen cloth types (i.e., flattening different types of T-shirtswhen only trained on rectangle cloths), and reach range (i.e., on cloths with dimensions larger than its reachrange).
Finally, we evaluate the algorithm’s performance on the real-world setup.
4.1 Metrics
The algorithm performance is measured by the final coverage, delta-coverage from initial coverage, and thenumber of interactions.
Final coverage measures the coverage at the end of an episode, whereas delta-coverageis final coverage minus initial coverage.
All our coverage statistics are normalized (i.e., divided by themaximum possible coverage of the cloth in a flattened configuration) and can be easily computed from atop-down camera, while the number of interactions is the episode length.
To evaluate our policy, we load atask from the testing task datasets then run the policy for 10 steps or until the policy predicts grasps on thefloor.
The metrics are averaged across all tasks in the testing datasets.
[Figure]
[Figure] [Figure]
Sim
[Figure]
few interactions step, while the quasi-static baselines never reach such a high coverage even with significantly more interaction steps.
This demonstrates the difficulty of unfolding from highly crumpled initial configurations and dynamic action’s superior efficiency in such challenging tasks.
4.2 Task Dataset Generation
Each task is specified by a cloth mesh, mass, stiffness, and initial configuration.
The cloth mesh is sampledfrom one of three types:
Normal Rect , which contains rectangular cloths with size within the reach range.
Edge lengths aresampled from [ 0 .
40m , 0 .
65m ]2.
Large Rect , which contains rectangular cloths with at least one edge larger than the reach range (0 .
70m).Otherwise, edge sizes are sampled from [ 0 .
40m , 0 .
75m ] , which means the shorter edge could still besmaller than the reach range.
Shirt , which contains a subset of shirts sampled from CLOTH3D’s [23] test split, all of which are resizedto be within the reach range.
These shirts include tank tops, crop tops, short and long sleeves.
The cloth mass is sampled from [ 0 . 2kg , 2 . 0kg ] and an internal stiffness from [ 0 . 85kg / s 2 , 0 .
95kg / s 2 ] .
Finally, the cloth’s initial configuration is varied by holding a randomly grasped the cloth at a random heightbetween [ 0 .
5m , 1 .
5m ] then dropping and allowing the cloth to settle (similar to [2] and tier-3 in [3]), resultingin a severely crumpled configuration.
To calculate the normalized coverage, we use the maximum possible coverage of the cloths in their flattenedconfigurations.
For rectangular cloths in simulation, the flattened configuration can be analytically calculatedusing the undeformed vertex positions of the grid mesh, which means the normalized coverage could still behigher if the cloth rests in a stretched position due to friction.
For shirts in simulation, we opted to calculateits maximum possible coverage as its outer surface area divided by 2, since a qualitatively flattened T-shirtmay not actually maximize coverage.
While this choice also makes it possible for the normalized coverageto be greater than 1, it will still preserve performance rankings.
For real world experiments, the flattenedconfigurations are manually set.
We emphasize the difficulty of our unfolding tasks, where cloths in Normal Rect, Large Rect, and Shirthave average initial coverages of 28.8%, 27.1%, and 46.4% in simulation respectively, and 26.1%, 33.6%, and24.0% in the real world respectively.
In contrast, cloths in simulation from [3] start at 77 .
6%, and42 . 0% for easy, medium, and hard tasks, while cloths in the real world from [16] starts at 71.4% and 68.4%on two different real-world trials.
Yet, the challenging cloth configurations found in our tasks (Fig. 5) arerealistic and prevalent in typical households.
In simulation, the policy is trained on 2000 rectangular cloths sampled evenly between Normal Rect and
Large Rect, and evaluated on 600 novel tasks split evenly between Normal Rect, Large Rect, and Shirt cloths.
In real, the simulation policy is deployed to collect real world experience on 150 Normal Rect cloth tasks,optimized on both simulation and real world data, then evaluated on 10 novel test tasks in each cloth type.
4.3 Approach Comparisons
• Quasi-static : [Pick & Place] and [Pick & Drag] are policies which uses a quasi-static “pick and place”(similar to Lee et al.
[2], visualized in Fig. 2b) and “pick and drag” (similar to Seita et al.
[24], with no lift
step compared to pick and place) primitive respectively.
[Stretch & Drag] is identical to [Pick & Drag] with
F li ng
Init
N o r m a l R ec t
( R e a l )
P &
P
F li ng
L a r g e R ec t ( R e a l )
P &
P
F li ng
Sh i r t ( R e a l )
P &
P
Step
Step0.
Init
Step
Step0.
Done 0.
Done
Done 0. Done 0.
Step
Step0.
FlingBot discovered through trial-and-error a two-arm corner and edge grasp when corners and edges are visible.
While the pick & place baseline discovered a similar strategy, its performance is inherently limited by quasi-static actions, requiring significantly more steps to achieve a final coverage lower than FlingBot’s.
an extra stretch step aftering picking, similar to FlingBot.
These baselines are implemented with dual-armset up, therefore provides the same physical reach range as our dual-arm fling system.
• Dynamic manipulation : [FlingBot] is our policy which predicts the optimal two-arm grasp locations for
a fixed stretching and flinging routine described in Sec. 3.
• Dynamic manipulation with fling speed prediction : [FlingBot-S] is identical to [FlingBot] with anadditional fling speed module, which predicts the fling speed within the range of [ 0 .
1ms − 1 , 1 .
0ms − 1 ]from visual input of the cloth after stretching and lifting.
This fling speed module is trained using DeepDeterministic Policy Gradients (DDPG) [25] on the delta-coverage rewards to maximize single-step returns(discount factor γ is set to 0), where the grasps are predicted by a converged and frozen [FlingBot] valuenetwork.
However, despite the cloth physical parameter variations in the training
and
testing dataset
Sec. 4.
, Tab. 1 and Fig
suggest that there aren’t significant performance gains for
[FlingBot
S]
over
[FlingBot]
within the cloth
parameter ranges
tested
(Sec.
These
results
justify [FlingBot]’s usage
of asingle fling speed, even for varying cloth mass, stiffness
and size.
Thus, we prefer the simpler [FlingBot]approach for comparisons with baselines.
286 • Dynamic manipulation with fling parameter regression :
[Fling-Reg] is identical to [FlingBot], butdirectly predicts the action parameters (cid:104) C x , C y , θ , w (cid:105) from visual inputs instead of exploiting the task’sequivariances.
Its policy is trained using DDPG on delta-coverage rewards to maximize single-step returns.
However, from Tab. 1, [Fling-Reg] completely fails to perform the task, demonstrating the advantage ofencoding inductive biases which leverage equivariances in the problem structure.
4.4 Results
Better Efficiency.
In this experiment, we compare the unfolding efficiency between quasi-static and dynamicactions.
From the Normal Rect column in Tab. 1, [FlingBot] increases the coverage of the cloth ( + 63 .
1%) bymore than two times that of [Pick&Place] and [Pick&Drag] baselines ( + 29 .
2%, and + 24 .
Additionally,from Fig. 4, [FlingBot] achieves over 80% within 3 interactions (simulation normal cloth), while the quasistatic baselines never reach such a high coverage even with significantly more interaction steps or with a stretching subroutine.
This demonstrates the difficulty of unfolding from highly crumpled initial configurationsand dynamic action’s superior efficiency in such challenging tasks.
Increased Reach Range.
In this experiment, we investigate these approaches’ performance on cloths whichhave dimensions larger than the robot arm’s reach range.
These tasks are not only challenging for quasi-staticbaselines, because the arms can’t manipulate the cloth at locations beyond its reach range, but also for ourflinging policies, because the arms can’t fully stretch or lift the cloth off the ground while flinging.
Despitethese challenges, [FlingBot] achieves 79 .
2% (Tab. 1, column Large Rect).
Compared to the quasi-staticbaselines, [FlingBot] increases the coverage by + 52 .
%, which is roughly twice that of the quasi-staticbaselines ( + 27 .
1%, + 24 .
8%, + 23 .
These results show how high-velocity actions could throw cloths toregions unreachable by quasi-static actions, allowing the system to be more flexible with extreme cloth sizes.
[Table]
Generalize to Unseen Cloth Types.
In this experi-
[Table]
ment, we investigate how well these approaches, trainedon only rectangular cloths, can generalize to nonrectangular cloths such as different types of T-shirts.
Qualitative real world (Fig. 5) and simulation (in supplementary materials) results suggest that our flingingpolicy has learned to grasp keypoints on cloth (i.e., corners, edges) when it sees them, or otherwise fling toreveal these features.
FlingBot’s generalization performance (93.3% in Tab. 1, column Shirt) to novel clothgeometries like shirts can be attributed to this strategy,since a cloth of any type can be unfolded by grabbing one of its edges, stretching, then flinging it in a directionperpendicular to the edge.
Through self-supervised exploration, our approach discovered grasping strategieswhich were manually designed in heuristic based prior works, while being more generalizable to differentcloth configurations and types.
Meanwhile, all quasi-static baselines exhibited worse cloth unfolding efficiency.
The Sim Shirts plot in Fig.
4 shows that our flinging policies take only 3 actions to reach their maximumcoverages, while the quasi-static baselines take upwards of 8 steps to reach lower maximum coverages.
[Table]
Evaluating Real-World Unfolding.
Finally, we finetuneand evaluate our simulation models from Tab. 1 with realworld experience on a pair of UR5 arms.
Task generation isautomated using the robot arms by randomly grasp the clothat height 1 m and drop it back on the workspace.
We usea 0 .
35m × 0 .
45m cloth for Normal Rect, a 0 .
40m × 0 .
70mbath towel for Large Rect, and a 0 .
45m × 0 .
55m T-shirtfor Shirt.
The system collected 257 experience steps over 150 cloth tasks for finetuning in total.
Theperformance is reported averaged over 10 test episodes, where real-world grasp errors are filtered out (see“Real World Failure Cases” below).
From Tab. 2, we report that our policy achieves over 80% on all clothtypes, which outperforms the quasi-static pick & place baseline by at least 40%.
Additionally, we report thatthe pre-finetune performance of FlingBot on Normal Cloths is 69.8%, justifying our decision to finetune to geta 12.1% improvement.
Overall, our flinging primitive takes a median time of 16 .
7s.
While the pick & placeprimitive only takes a median time of 8 . 8s, it is unable to reach the same coverage our flinging primitive, evenwhen taking many more interaction steps.
Both primitives incur additional overheads of 2 .
5s for preparing thetransformed observation batch and 2 .
9s for a routine which checks whether the cloth is stuck to the gripperafter the gripper is opened (details in supplementary materials).
Real World Failure Cases.
Grasp failures - when the policy picks a grasp on the cloth, but the gripper failsto pinch grasp the cloth - make up most of the failure cases in FlingBot’s real world system, and we discardepisodes where grasp failures occur.
Additionally, even on successful grasps, the gripper may hold the clothtight in its crumpled state, rendering all flings ineffective.
However, we do not discard of these episodes.
Wediscuss more of real world grasp failures and show simulation failure cases in the supplementary material.
Conclusion and Future Work
We proposed a dynamic fling motion primitive and a self-supervised learning algorithm for learning the graspparameters for the primitive of the cloth unfolding task.
The cloth unfolding policy is efficient, generalizable,and works with cloth sizes beyond the reach range of the system in both simulation and the real world.
Whilewe’ve demonstrated the effectiveness of dynamic actions for cloth unfolding, dynamic actions alone areinsufficient for more complex deformable object manipulation tasks, such as goal-conditioned folding.
Inthis direction, future work could explore learning in combined action spaces (i.e., dynamic and quasi-static,two-arm and single-arm, etc.)
for goal-conditioned cloth manipulation.