“Good Robot! Now Watch This!”: Repurposing Reinforcement Learning for Task-to-Task Transfer
Anonymous Author(s)
AfﬁliationAddress email
Abstract: Modern Reinforcement Learning (RL) algorithms are not sampleefﬁcient to train on multi-step tasks in complex domains, impeding their widerdeployment in the real world.
We address this problem by leveraging the insightthat RL models trained to complete one set of tasks can be re-purposed to completerelated tasks when given just a handful of demonstrations.
Based upon this insight,we propose See-SPOT-Run (SSR), a new computational approach to robot learning that enables a robot to complete a variety of real robot tasks in novel problemdomains without task-speciﬁc training.
SSR uses pretrained RL models to create vectors that represent model, task, and action relevance in demonstration andtest scenes.
SSR then compares these vectors via our Cycle Consistency Distance(CCD) metric to determine the next action to take.
SSR completes 58% more tasksteps and 20% more trials than a baseline few-shot learning method that requirestask-speciﬁc training.
SSR also achieves a four order of magnitude improvementin compute efﬁciency and a 20% to three order of magnitude improvement in sample efﬁciency compared to the baseline and to training RL models from scratch.
Toour knowledge, we are the ﬁrst to address multi-step tasks from demonstration ona real robot without task-speciﬁc training, where both the visual input and actionspace output are high dimensional.
Code is available in the supplement.
Introduction
[Figure]
The ability to quickly adapt deep Reinforcement Learning (RL) models to new tasks with just ahandful of demonstrations would dramatically enhance their practicality and applicability.
Repurposing previously-trained models for new tasks consumes several orders of magnitude fewer computational resources than training from scratch.
For our approach to repurposing RL models, wedraw inspiration from social imitation and learning in animals.
Certain animals, dogs, for example,have a strong capacity to imitate human performance of a novel task when trained with the “Do as I
Do” method [1, 2].
Imagine a dog named Spot that can be been trained with positive reinforcement(“Good Spot!”) to imitate a new action shown to her, such as fetching a speciﬁc object from a container, when prompted by a command (“Watch This!”)
When Spot imitates is the orientation
II + III.
See
SPOT
[Figure]
and the shape of the action [that is] attended to and reproduced” [2, 3].
Spot encodes what she sees inan internal representation, which she may then map onto her trained skillset in order to successfullyimitate a new task.
We observe that some deep RL models, like Spot the dog, already encode meaningful representations of a scene that can be mapped to a skillset.
We hypothesize that we can leverage theseencodings to solve new tasks from demonstration without task-speciﬁc learning , a concept we call
Reinforcement Learning Before Demonstration (RLBD, Fig. 1).
To evaluate our hypothesis, wepropose the See-SPOT-Run Framework for RLBD.
See-SPOT-Run (SSR, Fig. 2) rapidly adapts an ensemble of existing RL policies to new tasks byimitating demonstrations.
Intuitively, existing fully-trained and validated RL models are alreadyinvariant to certain changes in the scene within the tested circumstances.
We pass a known stateand a novel state to a preexisting model to create and then compare latent vectors in a search forembedded task-invariant knowledge to use in novel circumstances.
When the known state is froma novel task demonstration, and the novel state is from a live test scene, we can pinpoint latentvectors that map to actions in the test environment that are likely to progress towards completingthe novel demonstrated task.
Therefore, we apply See-SPOT-Run to imitate demonstrations of anew multi-step task (“Watch This!”) by matching demonstrated actions to candidate actions in a testenvironment.
In doing so, we not only enable few-shot imitation of unseen tasks but also providethe potential to vastly expand robotic skill sets at dramatically lower costs.
In our few-shot imitation experiments, we demonstrate See-SPOT-Run’s ability to complete a setof multi-step tasks, including block stacking, row making, block unstacking, and building a verticalsquare structure (see Fig. 1).
These tasks are an order of magnitude more challenging than typicalreach or limited grasp and place tasks [4] (see Sec 3).
To evaluate each task, we use RL policiestrained to complete different, and even conﬂicting (stacking vs unstacking) tasks, and provide twoexpert demonstrations collected in simulation.
In summary, our contributions are as follows:
Reinforcement Learning Before Demonstration (RLBD): We propose a concept to use pretrainedRL policies to accelerate robotic imitation learning.
We show that these policies generate Action-
State Embeddings (ASEs) from robot state observations, and that these embeddings can be repurposed to execute unseen tasks.
See-SPOT-Run (SSR): Our RLBD implementation that accomplishes both few-shot imitationand zero-shot sim-to-real domain transfer by matching successful demonstrated actions to thetest environment with a novel Cycle Consistency Distance (CCD) metric.3.
SSR achieves competitive task performance without the task-speciﬁc training required by priorwork, leading to a four order of magnitude improvement in compute efﬁciency and a 20% to
three order of magnitude improvement in sample efﬁciency for novel tasks when compared to thebaseline and to training RL models from scratch.
Related Work
Deep learning approaches have successfully completed a range of robotic tasks in visually-guidedmanipulation [5, 6] and navigation with high sample complexity [7, 8, 9, 10, 11, 12, 13].
VPG [6]achieved particularly efﬁcient RL training for decluttering tasks.
“Good Robot!”
[4] improved on
VPG’s RL sample efﬁciency and showed zero-shot sim-to-real transfer for multi-step block taskswith a risk of progress reversal [4], e.g. toppling a partially completed tower (see Sec. 3); however,it requires from-scratch task-speciﬁc training.
Another approach to tackling deep RL’s high sample complexity is through multi-task and metalearning [14, 15, 16, 17, 18, 19, 20] of general purpose task representations or policies that allowan agent to quickly adapt to novel tasks by ﬁne-tuning previously learned policies.
James et al.
[16]approach the few-shot learning problem by ﬁrst meta-learning a task embedding network from a setof training tasks, then computing a test task embedding from a few demonstrations; this task embedding is provided as input to a control policy.
Meanwhile, Finn et al.
[14] perform one-shot imitationon tasks seen during meta-training but evaluated on unseen objects.
While these approaches achievegood performance on one-step tasks, they do not consider multi-step tasks or progress reversal .
Alternately, few-shot inverse RL methods [21, 22, 23] aim to increase RL sample efﬁciency by learninga reward function from one or a few demonstrations, then using this reward function to supervise RLtraining.
Yet, Liu et al.
[21] and Singh et al.
[22] do not address learning multi-step tasks, which canrequire dense reward functions to learn effectively [4, 10].
Goo and Niekum [23], in turn, proposean IRL approach for multi-step tasks that requires time and compute-intensive training.
Zhu et al.[24] reviews Transfer Learning, and our work can be framed as Representation Transfer.
Cycle Consistency is a mechanism for comparing representations across domains [25, 26, 27].
Zhang et al.
[28] utilizes Cycle Consistency for cross domain dynamics in simulated locomotiontasks and sim-to-real transfer of a basic robot reaching task, showing that Cycle Consistency canmitigate domain shift for the same, single-step, task.
Our See-SPOT-Run framework, by contrast,completes both sim-to-real and known-to-novel task transfer with more complex multi-step tasksvia our own cycle consistency concept.
Behavior Cloning (BC), Imitation Learning (IL) [29, 30, 31, 32, 33, 34, 35], and IL from Observation(ILO) [31], are approaches to mimic the behaviors of agents.
Recent reviews cover IL [36, 37] and
ILO [38].
Jung and Kim [39] take a hybrid BC and IL approach to grasp and place one block onanother block.
GAIL [40] learns to imitate with inspiration from generative adversarial networks.
Others pursue distribution matching approaches [33, 25, 30].
However, these BC, IL, and ILO papersvariously require training on novel tasks, are evaluated only on simulated tasks, or do not consider multi-step tasks with a risk of progress reversal [4].
NTP [41] and NTG [42] learn task plans in ablocks environment by imitating videos, but require a perception pipeline to act.
TransporterNets [29] models multi-step IL tasks characterized by progress reversal as a sequenceof displacements via behavior cloning, with impressive results that scale from 1-1k demonstrationsper task.
However, TransporterNets makes no attempt at task-to-task transfer for different tasks.
Instead, it requires real-world, task-speciﬁc demonstrations and training for every unique task, sotheir approach is simultaneously much easier to solve (direct task transfer) and much more difﬁcultto adapt to new scenarios (due to task-speciﬁc training).
In comparison, SSR addresses the problemof few-shot imitation with no task-speciﬁc training.
Even with these limitations, TransporterNets isthe closest available method to SSR and consequently, we chose it as our baseline for comparison.
Preliminaries
As in “Good Robot!”
[4] and VPG [6], we investigate multi-step tasks by framing our problemas a Markov Decision Process (MDP) ( S, A, P, R ) , with state space S ; action space A ; transitionprobability function P : S ⇥ S ⇥ A !
R ; and reward function R : S ⇥ A !
R .
We make an MDP- 111 simplifying assumption where states equal observations, and that there is a discrete action space.
Attime step t , the agent observes state s t and chooses an action a t according to its policy ⇡ : S !
A ,leading to a new state s t +1 with probability P ( s t +1 | s t , a t ) .
Q-Learning is an RL method whosepurpose is to seek a Q function that maximizes R over time.
Q-Learning policies ⇡ select an action a t at time t by ﬁrst estimating the reward R with a function Q : S ⇥ A !
R , then selecting theaction that maximizes the expected reward: a t = ⇡ ( s t ) = argmax a 2 A Q ( s t , a ) .
Our See-SPOT-Run Framework experiments are pretrained with the Q-Learning-based SPOT-Qframework introduced by “Good Robot!”
SPOT-Q demonstrates the importance of prioritizingprogress reversal during RL training, and also effectively integrates ‘common-sense’ explorationconstraints into Q-learning.
Our RL policies are pretrained with the same exploration constraintsand reward functions R trial , R P as the best performing models in [4].
In our development, we makeuse of a task progress measure, P : S !
Z [0 ,p max ] as an integer indicating the number of successfultask steps; the task is complete when P ( s t ) =
p max .
Consequently, a progress reversal [4] occurswhen an action a t at a timestep t undoes previous task progress i.e. there exists an i 2 [0 , t ] suchthat P ( s t  i ) >
P ( s t +1 ) .
Next, we will introduce our proposed methods.
Methods
[Figure]
place hidden visible
SSR has three phases: (1) Embed , (2) Correspond , and (3) Act .
In phase (1) Embed , SSR takes a set of fullytrained policies, such as Q-functions, and converts them into embedding functions by extracting the latent per-action feature vectors embedded in the ﬁnal network layer immediately before Q-values are created (Fig. 2).
SSR runsthese embedding functions on every demonstration and test state to make a unique latent spacefor each (policy, demonstration) pair.
Phase (2) Correspond has two steps: (2a)
Match and (2b) Verify.
In (2a), SSR comparesthe embedded demonstration state to the embedded test state of each latent space, searchingfor the test action that most closely matches thesuccessful demonstration action.
It ﬁnds thisclosest match by minimizing the euclidean distance between the demonstration action’s embedding (feature vector) and each potential testaction’s embedding.
Since matches are madein the embedding space, the matched actionshould mirror the qualities of the demonstratedaction for the new task, and need not bear anyresemblance to the action the original preexisting policy would choose for its original task.
Intuitively, the high dimensionality of latent vectorsensures dissimilar actions do not collide, in a manner analogous to hash functions.
For example,we successfully choose reasonable action matches within policies and across states (Sec. 5), butmatches across policies were random and unsuccessful in small tests.
Ultimately, this matchingprocess yields a set of candidate actions, one for each (policy, demonstration) pair.
In (2b) Verify, SSR reverses the process, starting with each test candidate action, retrieving its embedding, and performing the match process in the demo scene to see how that action would performin the demo world.
The physical cartesian distance between the known ideal demo action and ournew rematch demo action in the action space is our cycle consistency distance metric (Fig. 3); intuitively, actions that minimize this distance are more likely to accomplish the demonstrated behavior.
Finally, in phase (3) Act , SSR selects the candidate action that minimizes the cycle consistencydistance and acts.
It repeats this entire process until task completion.
SSR thus conducts a comprehensive search over all (policy, demonstration) pairs, jointly selecting the latent space anddemonstration that are most relevant to the novel task as well as the action that will most likelyadvance task progress.
In Section 4.1, we will describe how we construct the embedding functions, then in Section 4.2, we will elaborate on how SSR uses these embedding functions to imitatedemonstrations, and provide additional intuition for our cycle consistency distance metric.
4.1 Generating Embedding Functions
We begin by pretraining task-speciﬁc SPOT-Q [4] policies ⇡ m : m = 1 ...M on M known tasks:
Recall that Q functions are deﬁned v a = Q ( s, a ) ; we reframe this as V = Q m ( s ) , where V containsscores for every action a 2 A pertaining to task m .
Thus, the Q-value of taking action a in state sis Q m ( s )[ a ] .
Finally, from each Q m , we strip off the last linear layer w m , a linear projection (denselayer) that maps a feature vector to a scalar Q-value, to obtain as many Embedding Functions 1 E m :
[Equation]
Evaluating each E m on an arbitrary state yields a latent space array H m , composed of one ActionState Embedding (ASE) h m,
a = H m [ a ] for each action space coordinate a 2 A , as in Fig.
4.2 Test-Time See-SPOT-Run Policy
In this section, we will deﬁne our test-time SSR policy, ⇡ ssr t , which observes the test state s t andselects an action a t for the robot to execute, as it attempts to complete the demonstrated test task.
Itsolves the imitation problem : approximating an unknown optimal live test task action a ⇤ t that willcomplete the next step of a novel, demonstrated task.
SSR uses information from N demonstrations
D : { D 1 , ..., D N } .
Each demo D n : { ( s demo , a demo ) 1 , ..., ( s demo , a demo ) T } is a sequence of StateAction pairs which maximize R at each task step t 2 T .(1)
Embed: We begin by observing the test state s t and progress p =
P
( s t )
and we consider the
N state-action pairs ( s demo ,n , a demo ,n ) with progress p from the N demonstrations 2 .
We use theseto compute latent space representations of each demonstration state and the test state with eachembedding function E m , yielding M ⇥ N demo latent space arrays H m,n , and M test latent spacearrays 3 H m,t :
[Equation]
(2) Correspond:
We ﬁnd candidate test actions (2a), then verify these candidate actions with ourcycle consistency distance metric (2b).
To ﬁnd a candidate action, we match the demo action to thetest state by selecting the minimum euclidean distance between the demo action embedding (ASE)and all ASE vectors in the test latent space.
We repeat this comparison for each latent space, yielding
M ⇥ N total candidate actions F m,n , one for each (policy, demonstration) pair.
Candidateactions are expressed as coordinates in the action space A , and can be used to index quantities suchas the latent spaces H , which have the same dimension as A , as detailed in the Fig.
2 caption.
[Equation]
We then verify each matched candidate test action in the demo scene using our cycle consistencydistance metric.
We verify, or match in reverse, the candidate action’s ASE H m,t [ F m,n ] to thecorresponding demonstration latent space H m,n , minimizing euclidean distance in the latent spaceas above.
We expect the veriﬁcation action a rematch to be physically close to the original demonstratedaction a demo ,n in the action space A .
Therefore, we deﬁne the cycle consistency distance C m,n asthe Cartesian distance between a demo ,n and a rematch .
[Equation]
The purpose of the cycle consistency distance C m,n is to consider other ASE vectors in the latentspace H m,n that might indicate that a candidate action F m,n is not an authentic match.
Alternatively,the rematch step can be interpreted as an assessment of how relevant a given latent space is to thetest state and task at hand.
While Section 4.1 refers to latent spaces as H m , we extend this notation here, using H m,n to designate the state from the n -th demonstration embedded with E m .
Similarly, H m,t is the test state embedded with E m .
(3) Act: We select and execute the candidate action with minimal Cycle Consistency Distance (Eq 8).
We also evaluate a baseline SSR without cycle consistency by skippingthe rematch phase and directly minimizing the L2 Consistency Distance (Eq.
[Equation]
[Equation]
This completes our deﬁnition of the SSR policy a ssr t = ⇡ ssr ( s t ) .
Every time an action a ssr t completes we collect a new observation s t +1 , running the SSR policy ⇡ ssr repeatedly until task progress
P ( s T ) reaches its maximum, p max , indicating that the task is complete.
We evaluate See-SPOTRun with L2 consistency distance ( SSR L2CD in Tables 1, 2, and 3) and cycle consistency distance ( SSR CCD in Tables 1, 2, and 3), as well as prior work, with few-shot experiments in
Sec. 5.
We will also discuss how our SSR Framework with CCD surpasses the other methods.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
l
[Figure]
ﬁ c i e n
[Figure]
l
[Figure]
Log Scale.
Larger bubbles are better.
See
Experiments
We outline our assessment metrics in Sec.
5.1,provide simulation results in Sec.
5.2, thencover our sim-to-real transfer results in
Sec.
We pretrain models with SPOT-Q [4](Sec. 3) with the workspace, commands andaction space deﬁned in “Good Robot!”
Implementation and Robot details are in Fig.
1and Appendix C.1.
5.1 Evaluation Metrics
Our metrics quantify the broad improvementsin task performance and reductions in the resources necessary to perform novel tasks.
Mostimportant, we consider critical efﬁciency measures that have not previously been evaluated inthe baselines, to motivate the broad range of useful applications for both RBLD and SSR.
Test Metrics evaluate how effectively the robot completes the test tasks, and higher is better: (1)
Trial Success Rate (Trials) is the percentage of multi-step tasks completed 100% successfully, andin many applications completing a task is a prerequisite to moving on to the next task.
(2) Action
Efﬁciency (Eff.) is the idealactual number of actions per trial, and more efﬁcient models will completetasks with fewer actions.
[6] Our ideal is 6 actions for all tasks except for rows, which is 4 actions.
[4](3) Progress (Prog.) is each trial’s maximum proportional within-task progress averaged over alltrials, e.g. a 3 of 4 block stack is 75%, to show capabilities that nearly complete trials.
(4) Recoveries(Recov.) is the percentage of trials in which there was a mistake such as a progress reversal that theagent was able to complete with a trial success, i.e. trial successes containing progress reversal reversal .
Higher recoveryrates reﬂect better robustness to perturbations and uncommon situations.
Cost Metrics delineate resources spent, and lower is better: (5) Train Steps is the number of neuralnetwork batch steps performed prior to executing on a novel test task.
Each individual experimentis run on one NVIDIA GeForce RTX 2080Ti GPU.
(6) Annotated Actions (Ann.
Actions) is thenumber of robot actions a t that have been annotated by either a human or scripted observer, whichare each fairly high cost activities in robot or human time [43, 44].
Efﬁciency Metrics evaluate trial success beneﬁts with respect to the cost metrics, and higher is better.
(7) Compute Efﬁciency
[Table]
1 is the amortized percentage of trials that can be completedfor every training batch step when completing a novel task.
We add one to the train steps denominator to prevent dividing by 0 with SSR.
(8) Sample Efﬁciency
[Table]
measures the increase in 256 trial success rate amortized over annotated actions on a test task.
5.2 Simulation Experiments
We pretrain RL policies on four tasks as in Fig.
1: stacking, row-making, unstacking, and 2x2vertical square.
At test time we perform four-fold cross-validation where the test task is given two demonstrations but no pretrained model, then the remaining three models predict actions for the untrained test task.
For the ﬁnetuned case, we pretrain each of the three models on two demonstrationsof the untrained test task for 333 steps each, for a total of 1k steps. We then apply SSR as describedin Sec. 4.
We additionally investigate a “Good Robot” ﬁnetuning baseline where we use one of theﬁnetuned models directly as the test-time policy, without applying SSR. To obtain our TransporterNet [29] baselines on their less challenging scenarios (Sec. 2), we collect two demos of each test task in their ravens framework and train TransporterNet on these demos for 40k iterations; meanwhile, for the pretrained case we ﬁne-tune their vertical block trianglemodel [29] on the novel task for 1k iterations (thus matching the overall training cost of our pretrained method).
We will examine overall performance on the test metrics, task-speciﬁc test metricsof particular interest, and then our efﬁciency metrics.
[Table]
[Table]
[Table]
Our results on the overall Test Metrics in Table 1 show SSR with CCD achieving 36% trial completion, 79% average progress, 41% action efﬁciency, and 29% recovery rate; L2 Correspondence
L2CD gets 13%, 67%, 30%, and 5%, respectively, which demonstrates the beneﬁt of evaluatingdemo embeddings H d with Cycle Consistency Correspondence CCD (Alg. 2); and TransporterNetgets 30%, 50%, 35%, and 8%, respectively, which highlights our method’s overall improvementover prior few-shot imitation work.
Our task-speciﬁc simulation results in Table 2 show SSR with CCD performs better than L2CDin all cases, and better than TransporterNets in all cases except unstacking.
The SSR worst casetrial success rate of 24% is 12x better than the TransporterNets worst case of 2%.
SSR completes66% of unstacking trials vs 86% for TransporterNets, and SSR completes 30% of rows vs 2% for TransporterNets.
This highlights a shortcoming of TransporterNets’ feature template displacementmatching approach which is not present in SSR.
A more detailed per-task simulation breakdown isin Appendix C.3.
Lastly, the ﬁnetuning experiments show that SSR with CCD is effective not only in a zero-trainingsetting, but also as an extension to few-shot model ﬁnetuning.
Compared to the baseline ﬁnetuned“Good Robot”, our SSR CCD Finetuned achieves 86% vs 80% average progress, 51% vs 35% trialcompletion, 35% vs 43% action efﬁciency, and 30% vs 16% recoveries.
This highlights the mipactof the SSR Test Time policy (Sec. B.3), as SSR with Finetuning and “Good Robot” ﬁnetuninguse the same base models and ﬁnetuning method.
Moreover, SSR with Finetuning also outperforms
TransporterNets with Pre-Training, achieving 86% vs 57% average progress, 51% vs 41% trial completion, 35% vs 36% action efﬁciency, and 30% vs 12% recoveries.
Efﬁciency Metrics : With CCD, SSR achieves 36% compute efﬁciency, which is roughly fourorders of magnitude better than TransporterNet and “Good Robot!”, which both have 8 e-
4% compute efﬁciency.
This is triple the 13% compute efﬁciency of L2CD, further underlining the effectiveness of our CCD metric.
SSR also achieves three orders of magnitude better sample efﬁciencywith CCD than ‘Good Robot!”
, at 3% vs. 2 e- 3% , while maintaining a much smaller, but still signiﬁcant, lead over TransporterNet at 2.5% sample efﬁciency, and L2CD at 1.1% sample efﬁciency.
Thesmaller gap is because, as an imitation method, TransporterNets is able to replicate behaviors withvery just a few annotated actions.
Even so, SSR maintains its lead over TransporterNets due to itssuperior trial completion rate.
Next we will examine our performance on the real robot experiments.
5.3 Real Robot Experiments
[Table] transfer SSR to a real robot using our models pretrained in simulation from Section results in Tab. 3.
All other aspects of themethod remain the same except task progress isrecorded by a different observer, since in simulation we read internal simulator states.
TransporterNets is not designed for sim to real transfer, and is thus not included here.
In real experiments SSR has an average of 40%of trials complete, 42% action efﬁciency, 84%progress, 35% rate of recoveries, 40% compute;which is very similar to our results in simulation at 36%, 41%, 79%, 29%, and 36% respectively.
For reference, “Good Robot” completed 100%of trials after 20k RL training actions and 60k training steps for both stacks and rows, resulting in0.0008% compute and 0.003% sample efﬁciency; but it scored on the train task, and is neither designed for nor capable of completing new test tasks on its own.
We examine per-task real SSR testmetrics in Appendix C.4.
Conclusion
See-SPOT-Run’s three to four order of magnitude sample efﬁciency and compute efﬁciency improvements show it can immediately and effectively begin making progress on a novel task in anovel domain with scarce demonstration examples.
By contrast, RL models such as “Good Robot!”typically start out with highly randomized exploration and no ability to complete a novel task, requiring inefﬁcient sampling and compute-intensive training to ﬁrst match and then beat SSR on agiven task.
Furthermore, although SSR performs 12x better than TransporterNets [29] in the worstcase sim trials, TransporterNets may scale better than SSR on some metrics as demonstrations areadded.
Even so, it would ultimately be more efﬁcient in the long run to develop methods wheredemonstrations on one multi-step task also improve performance and efﬁciency on the next task.
For these reasons, we suggest that future research investigate how RLBD, Embeddings, and SeeSPOT-Run might guide the exploration and augment the training of other learning methods such as
Reinforcement Learning [4, 6], Behavior Cloning, Imitation Learning, and Meta-Learning.
Successin these avenues would substantially improve multi-task learning efﬁciency.
Further improvementsto RLBD itself might include better correspondence metrics, automatically assessing task progress,or otherwise mitigating the need for a dense reward [23, 26].
In summary, we have introduced the concept of Reinforcement Learning Before Demonstration,implemented via the See-SPOT-Run framework.
Our framework achieves simultaneous few-shotimitation and sim-to-real transfer for multi-step tasks, such as block stacking and row-making, thatare prone to progress reversal .
See-SPOT-Run improves compute and sample efﬁciency by three tofour orders of magnitude vs. prior work by leveraging pretrained RL models to solve the imitationproblem on novel tasks.