Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapes
Anonymous Author(s)
AfﬁliationAddress email
Abstract: We study the problem of robotic stacking with objects of complexgeometry.
We propose a challenging and diverse set of such objects that was carefully designed to require strategies beyond a simple “pick-and-place” solution.
Our method is a reinforcement learning (RL) approach combined with visionbased interactive policy distillation and simulation-to-reality transfer.
Our learnedpolicies can efﬁciently handle multiple object combinations in the real world andexhibit a large variety of stacking skills.
In a large experimental study, we investigate what choices matter for learning such general vision-based agents insimulation, and what affects optimal transfer to the real robot.
We then leveragedata collected by such policies and improve upon them with ofﬂine RL.
Introduction
There has been a plethora of work on applying learning algorithms to solving difﬁcult real robotmanipulation problems in a general way with a large and diverse set of objects. However, the focusof existing work has primarily been on tasks like grasping [1, 2, 3, 4], which do not typically requirecomplex inter-object contact dynamics.
Manipulation tasks involving interaction of diverse objects,e.g. construction of simple structures, require skills that involve control strategies that are morecomplex and qualitatively different to simply grasping and placing such objects. As a step in this direction, we study whether, and how, we can tackle stacking of diverse objects inthe real world via a learned policy, using only information from RGB cameras and proprioception.
Several works have recently considered vision-based real-robot stacking [5, 6, 7, 8, 9, 10]. However, the focus has almost exclusively been on stacking cube-like objects (which does not requirereasoning about orientation, shape, or stack stability), or on a limited set of known irregular objectswith limited diversity [5]. Additionally, prior work reported low task success rates, pointing towardsthe difﬁculty of stacking even cuboids in the real world. In order to study stacking in a principledway, we propose a new benchmark, RGB-Stacking , see Figure 1, which consists of a set of 152procedurally-generated and 3D-printed rigid geometric objects.
These were carefully designed topose different degrees of grasping and stacking difﬁculty for a parallel gripper on a 4-DoF arm.
Thebenchmark is standardized and we release all relevant information for replicating it as supplementary material 1 .
What differentiates our work are two primary characteristics: our large variety ofobjects and our extensive evaluations. To highlight the challenges in stacking these objects, we instantiated two RGB-stacking tasks (withcorresponding versions in simulation and real world) designed to investigate a set of research questions. In the ﬁrst task we consider mastering stacking for a set of 5 speciﬁc combinations of objects.
These, shown in Figure 3, were chosen in terms of the challenges they present: precision in graspingand stacking, balancing, and using the top object as a tool to ﬂip a bottom object that has a slanted [Figure] face pointing up.
For this task we were interested to see whether it is possible to learn a singlevision-based policy that can reliably stack all 5 combinations in the real world.
For the second taskwe considered the challenge of learning general stacking strategies from a large set of training objects and test how well they transfer to held-out objects —to assess generalization.
In both cases, weﬁnd that using an agent trained in simulation is the most efﬁcient way to bootstrap data collectionfor ofﬂine RL.
Our contributions are as follows.
(a) We present and release a benchmark for stacking that featuresa diverse set of geometric objects and tens of thousands of possible stacks.
(b) We show that itis possible to obtain a purely learned vision-based policy that can stack multiple combinations ofobjects and can demonstrate a variety of stacking strategies for non-cuboid objects, emergent from
RL training.
(c) We present the framework we used to obtain our results without the need forhuman demonstrations.
Speciﬁcally, we train vision-based agents with domain randomization via aninteractive distillation approach—decoupling learning the required stacking skills with RL trainingfrom mapping the skills to perception with imitation.
We also ablate individual components of thispipeline, requiring more than 54,000 stacking attempts on real robots.
(d) Finally, we show that anofﬂine RL step based on data gathered on the real robots can boost performance when the data iscollected with sim-to-real policies; but doing so based on real episodes collected by a scripted agentwas worse than zero-shot sim-to-real.
Related Work
Robotic Stacking.
Several works have recently dealt with vision-based real-robot stacking eitherby learning a curriculum of the different stages of the task [6], by combining human demonstrationsand RL [7, 8], or by using some ﬂavor of sim-to-real transfer [7, 9, 10].
In most prior work, the focushas almost exclusively been on cube stacking. Furrer et al [5] deals with stacking 6 known stonesof different shapes.
These stones however all have fairly wide support, high friction, require similarpick-and-place strategies, and only 11 episodes were used for evaluation of the suggested method.
Incontrast, RGB-stacking provides a general, reproducible, and signiﬁcantly more difﬁcult benchmarkfor robotic manipulation.
It involves thousands of stacking combinations and our evaluations aresigniﬁcantly more extensive than in prior work.
Large-Scale Deep RL for Robot Manipulation.
Deep learning for robotic manipulation was inpart popularized by large-scale data collection for grasping in the real world [1, 2, 3, 4], a taskthat allows for automated evaluation and resetting.
However, these efforts considered problemswith short time horizons and limited dynamical effects (e.g. position-controlled robots with actionstaking up to a second).
As a result they often used simple Q-learning from collected data with directoptimization for action selection [3].
In contrast we consider velocity-based control at a rate of 20 Hzleading to long episodes (400 timesteps).
This is a more challenging scenario in which learningwith, e.g., QT-Opt [3] from recorded data alone would exhibit problems due to wrongly-estimated
Q-values [11].
Additionally, large-scale RL-based learning in the real world can be difﬁcult toreproduce as the entire real-world process (which can last months [2, 3]) is a function of the initialconditions.
In contrast, our simulation-based training is inherently reproducible.
In-hand object [Figure]
Imitation Learning and (Ofﬂine) Reinforcement Learning.
For training in simulation, and subsequent training of sim-to-real policies, we use techniques from off-policy RL followed by an interactive imitation learning approach.
For training in simulation we consider RL with full stateinformation, in order to avoid problems with partial observability.
In this setting we use the MPOalgorithm [14]; a sample efﬁcient, state-of-the-art, off-policy actor-critic method.
Initial training is then followed by our pipeline for obtaining a (general) vision policy via a DAgger [15]style interactive imitation learning approach to distill the MPO policy.
We found this to signiﬁcantly outperform Behavior Cloning [16] / policy distillation [17]; due to the fact that DAgger-styletraining provides corrections for mistakes of the vision-based policy.
We use Domain Randomization [18, 19, 20, 21, 22, 23] to obtain good sim-to-real transfer in the absence of real-worlddata; more details on this are given in the supplementary.
Finally, when improving policies withofﬂine RL from real data we consider: i) a ﬁltered behavior cloning loss that is equivalent to CRRexp [24] but with data from a single policy source (analogous to work on combined ofﬂine andonline RL [25, 26]); ii) Behavior Cloning based on only successful trajectories.
Both allow forstable ofﬂine learning without the problems standard RL algorithms exhibit in this setting [11].
The RGB-Stacking Benchmark
Despite stacking being addressed as a robotic learning task in prior work, previous approaches havebeen limited to a reduced set of objects, typically cubical in shape.
We focus on the problem ofstacking a variety of objects, characterized by different shapes.
This is obtained by deﬁning a parametric family of RGB-objects .
The design principle is to vary the grasp and stack affordances ofthese objects for a parallel gripper.
Our choices signiﬁcantly change the difﬁculty of the stackingtask by requiring an agent to exhibit behaviors that go beyond a simple pick-and-place [27] strategy.
We do so while keeping the beneﬁts of automated learning and evaluation as in [28, 2].
In
Appendix A, we analyze the difﬁculty of the task in various ways.
We qualitatively evaluate thegrasping affordance based on general principles on force closure and object funnels [29], and quantitatively based on the Ferrari-Canny grasp metric [30, 31].
Similarly, we qualitatively depict thestacking affordance (Figure 2), which is varied by having bottom objects, whose top ﬂat surfacesdiffer in area, shape, and orientation.
We also quantitatively evaluate both affordances by evaluatingthe stacking performance of human teleoperators in simulation, and of a carefully scripted agent.
3.1 The RGB-Objects Family
Our objects are all obtained by applying a deformation to a seed cube, a 2D vertical extrusion of aplanar square.
We deﬁned 4 major axes of deformation, illustrated in Figure 2, resulting in differentshapes.
These shapes can be thought of as the vertical extrusion of a 2D shape.
The Polygon Axistransforms the planar square into a regular polygon.
The Trapezoid Axis morphs the planar squareto an isosceles trapezoid.
The Parallelogram Axis changes the orientation of the extrusion axis,from vertical to progressively more slanted axes.
Lastly, the Rectangle Axis uniformly scales theobject along the x, y or z-axis.
These deformations and their combinations form a parametric family of objects.
We obtain the ﬁnalset of objects by uniformly sampling, for both the major axes and all pairwise combinations , 8objects between the seed object and a maximally deformed object.
Figure 1(d) shows a subset ofthese in the real world, and Appendix A contains a complete depiction.
3.2 RGB-Stacking Tasks
The RGB-stacking tasks we tackle in this paper involve three objects in a basket in front of therobot.
Those are colored R ed, G reen, and B lue to signal to a vision-based agent which one shouldbe stacked on top of which, and which one is just a distractor.
In all our experiments, red is assignedto the top object, blue to the bottom object, and green to the distractor.
We note that the role of thelatter is not just to distract visually but also to serve as an obstacle during stacking.
A successfulstack is achieved when the red object is above the blue one and their centroids are vertically alignedwithin some thresholds.
A detailed description can be found in Appendix B.
3.2.1 Skill Mastery on 5 Speciﬁc Triplets
The ﬁrst task involves 5 RGB-object triplets, with each object in a triplet being from a major axis.
The triplets were chosen to have different degrees of difﬁculty for a stacking agent.
The task is toachieve skill mastery on these 5 ﬁxed combinations, shown in Figure 3, with a single vision-basedagent.
We explain the challenges for each triplet in a loosely descending order of difﬁculty:
T r i p l e t
T r i p l e t T r i p l e t T r i p l e t
T r i p l e t
All the 5 stacks shown here are performed by a single agent trained solely in simulation with domain randomization and transferred to the real robot in a zero-shot fashion. Each triplet poses its own unique challenges to the agent: Triplet 1 requires a precise grasp of the top object; Triplet 2 often requires the top object to be used as a tool to ﬂip the bottom object before stacking; Triplet 3 requires balancing; Triplet 4 requires preci- sion stacking (the object centroids need to align); and the top object of Triplet 5 can easily roll off if not stacked gently.
Triplet 1.
The main challenge is grasping the top object, a grasp that involvesthe gripper closing on the slanted sideswill fail.
The bottom object also provides difﬁculty, as it has a limited stacking surface and can easily roll.
Triplet 2.
In this triplet, the bottom object has sloped sides and may need tobe reoriented by using the the top object as a tool before stacking.
Triplet 3.
The challenge with thistriplet is to have a secure central graspfor the elongated object and balance iton top of the slanted bottom object.
Triplet 4.
An easy triplet, it has rectangular prisms for both red and blue objects; the challenge is primarily to aligntheir centroids, required for a stack tobe considered successful.
Triplet 5.
In this triplet, the top objectcan easily roll off the bottom object asit has a large number ( 10 ) of faces andis nearly cylindrical.
3.2.2 Skill Generalization
The second RGB-stacking task we are tackling is a transfer task.
For this we designate the axes ofall pairwise combined deformations (see Appendix A for details) as the training axes and the onesof single deformation—the major axes above—as the held-out axes .
Based on these, we created atraining object set , which consists of 103 different shapes, and a held-out set (containing 4 axes ofdeformation), containing 40 shapes.
The 5 ﬁxed triplets chosen for the previous task belong to theheld-out axes and ﬁnal performance is evaluated based on them.
Stacking a Diverse Set of Objects in the Real World with a General Policy
Since we have access to a simulator with a reasonably accurate model of our real robot and its cell,our approach is to solve the RGB-stacking tasks in three stages: 1) RL training of an expert insimulation using state information, 2) imitation of such an RL expert with a vision-based policy insimulation with domain randomization for both visual appearance and dynamics, and 3) a policyimprovement step using data gathered on the real robots.
Explicitly decoupling these steps allowsfor faster and cheaper iteration by enabling parallel experimentation and tuning for all three stages.
Notation.
We consider the Markov Decision Process (MDP) in which a policy π ( a | s ) —a probability distribution over actions a ∈ A that is conditioned on states s ∈ S —acts to maximize thediscounted sum of rewards, with sparse rewards r ( s, a ) and discount factor γ ∈ (0 , 1] .
We use
Q π ( s, a ) =
E ρ π
[ (cid:80) ∞ t =
γ t r ( s t , a t ) | s 0 =
s, a 0 = a ] to denote the Q-function, where ρ π is the trajectory distribution induced by π , and A π ( s, a ) =
Q π ( s, a ) − E a (cid:48) ∼ π
[ Q ( s, a (cid:48) )
] for the advantage whenexecuting π .
In the real world, we cannot assume access to full state information (e.g. object posesand labels), so both the policy and advantage are deﬁned based on observations o
( s ) ∈
O (actuatorpositions of the robot and two camera images) as π ( a | o ( s )) and A π ( o ( s ) , a ) , respectively.
4.1 Training Expert Policies from State Features in Simulation
We train expert policies via off-policy RL in simulation with a shaped reward.
We used the MPOalgorithm [14] for this purpose (details in Appendix D), but any RL algorithm could have been usedin this stage.
We found training directly from state s to be signiﬁcantly faster than training fromvision and thus trained expert policies π e ( a | s, y ) —where y denotes a triplet number in { 1 , . . .
, 5 }for the Skill Mastery task, or the parameters of object deformation for the Skill Generalization task.
4. Interactive Imitation Learning of a Vision-Based Policy for Sim-to-Real Transfer
We use imitation in simulation to distill the state-based expert(s) into a single “student” vision-basedpolicy π vis θ ( a | o ( s )) that can be executed in the real world, where accurate object state is unavailable,and that is designed for maximizing sim-to-real transfer using components described in Section 5.2.
We could attempt to train this policy via supervised Behavior Cloning (BC) [16] on a dataset oftrajectories from the teacher, but we ﬁnd this suffers from covariate shift, where the student visitsdifferent states than the teacher (in which it fails).
We instead perform DAgger-style [15] interactive imitation learning (IIL) of teacher actions on states from trajectories obtained by executing thestudent policy.
We refer to this as IIL, in which we perform optimization alongside data-collection(rather than alternating collection and supervised learning as in [15]).
Speciﬁcally, our student policyπ vis θ is continuously executed in a heavily domain randomized simulation environment (see appendixfor details), collecting trajectories stored in a replay buffer D vis .
The student is trained to maximize:
[Equation]
where ( y, τ ) is a pair of object information y and trajectory τ
= ( s 0 , a 0 , s 1 , . . .
, s T ) .
Note that if wereplace the collected data D vis with expert data D e from executing the teacher π e , then we recoverstandard BC but with inﬁnite data.
The form of the teacher and the data above depends on the task.
Skill Mastery.
We have access to 5 teachers indexed by the object triplet number y ∈ { 1 , . . .
Each teacher { π e ( a | s, y = 1) , . . . , π e ( a | s, y = 5) } is independently trained only on the respectivetriplet y .
The student collects trajectories on all triplets, chosen at random for every episode.
Skill Generalization.
We use the same technique for the generalization task, except that now wehave access to a single general teacher π e ( ·| s, y ) .
This generalist is conditioned on parameters y ofobject deformation (see Figure 2) and can specialize to different objects.
During IIL, trajectories in
D vis are from the training objects, as the test objects will not be seen until ﬁnal evaluation.
4.3 One-Step Ofﬂine Policy Improvement for Sim-to-Real Policies
Once we obtained a sim-to-real policy, we consider improving its behavior on the real robots.
Weperform a single step of policy improvement in which we: 1) deploy π vis θ on the robots to collecta dataset D real , 2) train a new policy π imp θ ofﬂine, using a ﬁltering function f
( s, a, τ ) , and 3) deploy the improved policy on the robot again for evaluation (1-3 could be iterated to obtain a policyimprovement loop).
The ofﬂine training step aims to maximize the following objective:
[Equation]
We use the ﬁlter f
( s, a, τ ) =
exp( A π imp θ ( o ( s ) , a ) /α ) with temperature α (a hyperparameter) andlearned Q-function (to calculate advantage), referred to as CRR-IMP, which recovers the CRRexp [24]/AWAC [25] objectives, but with data from the deployed policy only.
Unlike in Section 4.1,we here use a binary, sparse, reward for success per step.
We also consider f ( s, a, τ ) =
r ( s T ) ,referred to as BC-IMP, which corresponds to BC on successful trajectories only (stack in last frame).
Experiments: Solving RGB-Stacking
We outline our efforts to solve the two stacking tasks we proposed in Section 3.2.
In both tasks weevaluate our agents on 5 ﬁxed object triplets, and we assume access to unlimited simulated data butonly to a ﬁxed budget ( < 100 000 ) of real episodes.
In Skill Mastery we have access to the ﬁxedobject triplets during all stages of training.
In Skill Generalization our agents only have access tothe training object set; the 5 ﬁxed triplets are now held out both in simulation and the real world.
In all experiments we use a standardized evaluation protocol: the positions of the objects are randomized at the beginning of each episode, and the agent has 20 seconds at 20 Hz to stack the redobject on the blue one.
We deﬁne task success as having a stack at the end of the episode.
Thisdeﬁnition was chosen to exclude cases in which a stack is brieﬂy achieved incidentally in the middleof an episode when the red object is above the blue one but subsequently falls off.
In simulation,we evaluate on the training object set by running each policy on 5000 random object triplets for 2episodes each.
For the 5 ﬁxed test triplets, we evaluate each policy for 1000 episodes per triplet.
Inthe real world, we only evaluate on the test triplets, as evaluating on a sufﬁciently large sample oftraining objects to obtain statistics is impractical.
We use one robot per triplet (evaluating for 200episodes), for a total of 1000 episodes per policy.
Mean performance is calculated by averaging thesuccess rate across the triplets.
For state-based policies and vision-based policies trained on real datawe report the average across 2 training runs (i.e. random seeds), and for the vision-based policiesdistilled from a teacher we report the average across 4 runs: 2 distillation runs for each of the 2teacher seeds.
Results for all runs individually are in Appendix E.
5.1 Evaluation on the Simulation Stacking Benchmark
We ﬁrst evaluate several baselines on our simulated benchmark task.
We are interested in the following questions: (a) How difﬁcult is the task in simulation?
(b) Would it be preferable to train directlyfrom vision?
(c) How does interactive imitation and the design choices for it affect performancecompared to the state-based teacher?
To assess the task difﬁculty, we tuned a scripted agent with access to the object positions and evaluated it on both our training set and our test triplets.
We also hired 4 individuals with no relationto the research team to attempt the sim task via teleoperation and a game pad for 846 episodes.
Fordetails on these indicative baselines, see
Appendix C.
These demonstrate the difﬁculty of the taskas in both cases the success was lower than 50% (see Table 1).
Next, we evaluate our benchmark tasks in the dense reward setting with MPO from state.
A ﬁrstﬁnding was that the state agents, which are given the full pose of each of the three objects, learn thetask 10 times faster than the vision agent; as illustrated in Appendix D, using a vision agent to trainfrom scratch on the robot would be impractical for these tasks.
The MPO state-based agents obtain79 .
3% on the test triples in the skill mastery setting and 68 . 8% on the training objects in the skillgeneralization setting, signiﬁcantly outperforming the scripted agent.
However, generalization fromtraining to test objects is only slightly higher than the scripted agent’s performance ( 47 . 8% ).
This isbecause these state-based agents are conditioned on object parameters, which are out of distributionfor the test triplets.
After obtaining vision agents via IIL-s2r we observe (Table 1, bottom) thatalthough the training set performance drops, performance improves for the test triplets, as the agentscan now generalize from visual cues to make inferences about the shape.
We next compare the interactive imitation learning setting (IIL) to learning from teacher trajectories with a behavior cloning loss (BC).
Rather than explicitly constructing a ﬁxed size dataset, wegenerate the data on-the-ﬂy by executing the teacher continuously during training.
As shown in
Table 1, learning from data generated by the student (interactive imitation) is crucial:
IIL performedsigniﬁcantly better than the alternative in both task settings.
5.2 Evaluation on the Real-World Stacking Benchmark
We then investigate the following questions related to RGB-stacking in the real world: (a) Is itpossible to solve the challenging tasks on real robots with a single vision policy, and is that betterthan a scripted baseline tuned for the test triplets?
(b) How well does zero-shot transfer of simtrained agents solve the tasks, and which of our decisions were important for zero-shot performance?
(c) Do we get a single-step improvement with BC-IMP and CRR-IMP from using data collected in
the real world?
(d)
And how does the data distribution affect the performance of these algorithms?
[Table] [Table]
As described in Section 4.2, we distill a vision-based policy from a single or multiple state-basedexperts in simulation.
These are 5 specialists for the Skill Mastery task and a single generalist forthe Skill Generalization task.
We can then directly execute this distilled vision policy on the realrobots.
As discussed previously, this decoupling allowed us to quickly iterate and investigate whataids sim-to-real transfer for these tasks.
We compare, in Table 2, the simulation performance andthe zero-shot sim-to-real transfer performance when ablating these various choices for our method.
Policies were trained up to 1 million learner steps, and the sim-to-real policy was selected to be theone with the highest performance in the training setting for each task.
As our policies are stochastic,for inference we have the option to execute a random sample from the action distribution or itsmode (i.e. the action with highest probability).
We execute the stochastic policy in simulation andthe deterministic mode of the policy on the real robots, unless otherwise speciﬁed.
We are able toachieve high performance of 67.9% on the Skill Mastery task and 51.9% on the Skill Generalizationtask in the real world.
Qualitatively, we also see in Figure 3 that we can, with a single vision-basedagent, show a diverse set of skills to address the challenges needed to solve the task for the 5 speciﬁctriplets we have chosen.
Videos of the behavior are in the supplementary.
Ablations.
We ablate different algorithmic choices in Table 2 (extended table in the supplementary).
One of our decisions was on the BC loss for distillation, which can either be thenegative log-likelihood or the mean squared error (MSE) , which respectively correspond to− log π vis θ ( a e | o ( s t )) or E a ∼ π vis θ [ || a − a e || 2 ] , where a e are actions from the teacher (the two are notequivalent for the Gaussian case as the policy variance is state dependent).
We also decided to usea hybrid action space, where the 3D Cartesian and angular velocities are modeled as continuousactions (Gaussian), but we have a binary gripper action (Bernoulli).
This applies to both statebased and vision-based policies.
This choice seemed to be the most important factor for sim-to-realtransfer success.
However, note that the effects are only visible when transferring (correspondingsimulation performance does not change).
For the Skill Generalization task, the state-based expertis conditioned on the object parameters y .
However, the object parameters are not passed to π vis θ —the agent needs to infer physical object properties from image observations.
We also chose to usea Transformer model [32] after the ResNet vision encoder for the policy’s network architecture,which gives the agent access to temporal information with an attention mechanism.
The Transformer does not have a signiﬁcant effect on Skill Mastery .
However, it seems to be able to utilizethe skill variety of a state-based expert conditioned on the object parameters a lot more successfully,when compared to an MLP model with the same number of weights.
This is clear both in simulationand the real world (and we hence used it in all experiments).
Random action execution delaysof 0 , 1 or 2 timesteps, and standard image augmentation (random color perturbations and imagetranslations) did not seem to affect transfer but we decided to include them in our training procedureto increase our agents’ robustness to natural perturbations.
[Table]
Our sim-to-real approach also outperforms a scripted agent tuned on the test triplets, as shownon Table 3.
For the Skill Mastery task, we further improve results via ofﬂine RL with BC (BCIMP) on 32 651 successful real episodes collected by the scripted agent and with CRR (CRR-IMP)on a total of 67 446 episodes, which included the unsuccessful ones collected by the same agent.
Hyperparameters and architecture choices were selected after such an investigation in simulation.
Asevident in the table, learning from scripted agent data is not better than sim-to-real.
Qualitatively, thepolicies learned seem to exhibit the same “robotic” movements as the scripted policy that generatedthe data they were trained on.
However, a single policy improvement step based on data collectedby a sim-to-real agent with an average success of 69.7%: 85 213 episodes ( 58 979 successful) yields74.6% (BC-IMP) and 81.6% (CRR-IMP), resulting in policies with remarkable stacking consistency.
As data collection for the training object set is particularly time-consuming, we only did a singlecollection ( 38 446 episodes, 37 .
4% of which were successful) with an earlier iteration of a simto-real agent for the Skill Generalization task (performance of 32 . 6% when evaluated on the testtriplets).
As before we trained BC-IMP and CRR-IMP on this data and again observed improvedperformance on the test triplets, this time using only the episodes collected from the training objectset.
Note that even though the suboptimal sim-to-real agent trained on the training set of objects wasperforming worse than the scripted agent, and the data in this case was more limited and of differentobjects, BC-IMP and CRR-IMP trained on this more diverse data lead to a signiﬁcant improvement.
Finally, Table 3 showcases the variation in difﬁculty of the 5 triplets: all methods perform best ontriplets 4 and 5, as these can sometimes be solved with a stereotypical “pick and place” behavior.
Incontrast, triplets 1, 2, and 3 each require specialized, advanced, strategies as indicated by the largegap between sim-to-real performance and scripted performance in the Skill Mastery task.
Conclusion
We studied the problem of vision-based stacking with a large variety of geometric objects that require a diverse set of stacking strategies.
We propose a benchmark for studying this problem in aprincipled way, and make signiﬁcant progress in solving two tasks involving both skill mastery onspeciﬁc objects and generalization across them.
We use simulation-trained agents to collect datain the real world, which in turn can be used for further performance improvement with ofﬂine RL.
Our best agent is a single vision-based policy that is capable of a variety of stacking strategies andachieves high performance.
Finally, we provide thorough real-world evaluations and discuss whatis important for solving our tasks.
Despite this success, our benchmark still poses many open challenges as, for example, shown by the gap between Skill Mastery and Skill Generalization results andwe hope that it can help development of new methods for learning general policies (e.g. by furtheradaptation at robot deployment time).