Efﬁcient Adaptation for End-to-End Vision-Based Robotic Manipulation
32% → 63% 49% → 66% 50% → 90% 75% → 93% 43% → 98% https://colordesigner.io/convert/hextohsv (HSV) 86%
Abstract —One of the great promises of robot learning systems is that they will be able to learn from their mistakes and continuously adapt to ever-changing environments. Despite this potential, most of the robot learning systems today are deployed as a ﬁxed policy and they are not being adapted after their deployment.
In this paper, we present a method and empirical evidence towards a robot learning framework that facilitates continuous adaption.
In particular, we demonstrate how to adapt vision-based robotic manipulation policies to new variations by ﬁne-tuning via off-policy reinforcement learning, including changes in background, object shape and appearance, lighting conditions, and robot morphology. Further, this adaptation uses less than 0.2% of the data necessary to learn the task from scratch.
We ﬁnd that the simple approach of ﬁne-tuning pretrained policies leads to substantial performance gains over the course of ﬁne-tuning, and that pre-training via RL is essential: training from scratch or adapting from supervised ImageNet features are both unsuccessful with such small amounts of data.
We also ﬁnd that these positive results hold in a limited continual learning setting, in which we repeatedly ﬁne-tune a single lineage of policies using data from a succession of new tasks.
Our empirical conclusions are consistently supported by experiments on simulated manipulation tasks, and by 52 unique ﬁne-tuning experiments on a real robotic grasping system pretrained on 580,000 grasps.
For video results and an overview of the methods and experiments in this study, see the project website at https://sites.google.com/view/adapt-on-the-ﬂy-vlrr2020.
I. I NTRODUCTION
The ability to constantly learn, adapt, and evolve is arguably one of the most important properties of an intelligent agent prepared to exist in the real world.
Similarly, our robots should be able to continuously learn and adapt throughout their lifetime to the ever-changing environments that they are deployed in.
This is a widely recognized requirement.
In fact, there is an entire academic sub-ﬁeld of lifelong learning [65] that is interested in the problem of agents that never stop learning.
Despite the wide interest in this ability, most of the intelligent agents deployed today are not tested for their adaptation capabilities.
Even though techniques such as reinforcement learning theoretically provide the ability to perpetually learn from trial and error, this is not how they are typically evaluated.
Instead, the predominant method of acquiring a new task with reinforcement learning is to initialize a policy from scratch, collect entirely new data in a stationary environment, and evaluate a static policy that was trained with this data.
This static paradigm does not evaluate the robot’s capability to adapt.
It also traps robotic reinforcement learning in the worst-case regime for sample efﬁciency: the cost to acquire a new task is dominated by sample efﬁciency of the learning algorithm and the complexity of the task, as reﬂected in cost of acquiring diverse task data starting from na ¨ ıve (e.g. random) exploration.
Most machine learning models successfully deployed in the real world, such as those used for computer vision and natural language processing (NLP) do not live in this regime.
For instance, the predominant method of acquiring a new computer vision task is to start learning the new task with a pre-trained model for a related task, acquired from a pre-collected data set, and ﬁne-tune that model to achieve the new task [11, 24, 10].
This changes the sample efﬁciency regime of the learning process from one which is dominated by task complexity to one that is dominated by task novelty , i.e. the difference between the new task and the task on which the model was pre-trained.
While a number of works have studied how to use pre-trained ImageNet [9] features for robotics [68, 25, 34], there are remarkably few works that study how to adapt motor skills themselves.
Our work attempts to bridge this gap.
We adapt an image-based grasping policy to changes in background, object shape and appearance, lighting conditions, and robot morphology and kinematics, while using less than 0.2% of the data necessary to learn the same task from scratch (see Fig. 1).
Our results, supported by simulation and extensive real-world experiments, indicate that a pre-adaptation policy acquired for a task using reinforcement learning can be used to acquire policies for nearby tasks using very little new data and a simple update procedure. Furthermore, we ﬁnd that this approach of adapting pre-trained policies with off-policy [Figure]
Explore [Figure] reinforcement learning (RL) leads to substantial improvements over the course of ﬁne-tuning, and that pre-training via RL is essential: it signiﬁcantly outperforms conventional pre-training techniques using supervised learning on task-agnostic datasets.
We believe this simple adaptation scheme provides a promising solution for creating a lifelong learning robotic agent, and show this potential using a simple continual learning experiment.
To our knowledge, this work is the ﬁrst to demonstrate that simple ﬁne-tuning of off-policy reinforcement learning can successfully adapt to substantial task, robot, and environment variations which were not present in the original training distribution ( i .
e . off-distribution).
II. R ELATED W ORK
We consider how we might transfer knowledge for efﬁcient learning in new conditions [63, 46, 61], a widely-studied problem particularly outside of the robotics domain [11, 24, 10, 7, 51].
Prior works in robotics have considered how we might transfer information from models trained with supervised learning on ImageNet [9] by ﬁne-tuning [36, 13, 17, 49] or other means [60, 20].
Our experiments show that transfer from pre-trained conditions is signiﬁcantly more successful than transfer from ImageNet.
Other works have leveraged experience in simulation [56, 66, 57, 62, 45, 55, 48, 22, 19] or representations learned with auxiliary losses [53, 39, 58] for effective transfer.
While successful, these approaches either require signiﬁcant engineering effort to construct an appropriate simulation or signiﬁcant supervision.
Most relevantly, recent work in model-based RL has used predictive models for fast transfer to new experimental set-ups [4, 18], i .
e .
by ﬁne-tuning predictive models [8], via online search of a prelearned representation of the space models, policies, or highlevel skills [-(Chatzilygeroudis et al., 
 ; Cully et al., 2015; Kaushik et al., 2020; Merel et al., 2019),-][5, 6, 30, 38], or by learning physics simulation parameters from real data [52, 28].
We show how ﬁne-tuning is successful with a model-free RL approach, and show how a state-of-the-art grasping system can be adapted to new conditions.
III. A V ERY S IMPLE F INE -T UNING M ETHOD
We deﬁne then evaluate a simple technique for ofﬂine ﬁnetuning.
Our experiments model an “on the job” adaptation scenario, where a robot is initially trained to perform a general task (in our case, grasping diverse objects), and then the conditions of the task change in a drastic and substantial way as the robot performs the task, e .
g .
through the introduction of signiﬁcantly brighter lighting, or a peculiar and unexpected type of object.
The robot must adapt to this change quickly in order to recover a proﬁcient policy.
Handling these changes reﬂects what we expect to be a common requirement of reinforcement learning policies deployed in the real world: since an RL policy can learn from all of the experience that it has collected, there is no need to separate learning into clearly distinct training and deployment phases.
Instead, it is likely desirable to allow the policy to simply continue learning “on the job” so as to adapt to these changes.
A. The Method
We deﬁne a very simple ﬁne-tuning procedure for off-policy
RL, as follows (Fig. 2).
First, we (1) pre-train a general grasping policy, as describe in Appendix A-A and [29].
To ﬁne-tune a policy onto a new target task, we (2) use the pre-trained policy to collect an exploration dataset of attempts on the target task; then (3) initialize the same off-policy reinforcement learning algorithm which was used for pre-training (QT-Opt, in our case) with the parameters of the pre-trained policy, and both the target task and base task datasets 1 as the data sources ( e .
g .
replay buffers); we then (4) update the policy with this training algorithm, using a reduced learning rate, and sampling training examples with equal probability from the base and target task datasets, for some number of update steps.
Finally, we (5) evaluate the ﬁne-tuned policy on the target task.
Our method is ofﬂine, i .
e .
it uses a single dataset of target task attempts, and requires no robot interaction after initial dataset collection to compute a ﬁne-tuned policy, which may then be deployed onto a robot.
B. Evaluating ofﬂine ﬁne-tuning for real-world grasping
We now turn our attention to how to evaluating this simple method’s effectiveness as an adaptation procedure for end-toend robot learning, and perhaps continual learning.
Our goal is to determine whether the method is sample efﬁcient, whether it works over a broad range of possible variations, and to determine whether it performs better than simpler ways of acquiring the target tasks.
With this goal in mind, we conduct a large panel of ablation experiments experiments on a real 7 DoF Kuka arm.
These experiments evaluate the performance of our method across the diverse range of Challenge Tasks (See Appendix A) and a continuum of target task dataset sizes, and compare this performance to two comparison methods. [Table]
The experiments are very challenging.
The Transparent Bottles task in particular presents a major challenge to most grasping systems: the transparent bottles generally confuse depth-based sensors and, especially in cluttered bins, require the robot to singulate individual items and position the gripper in the right orientation for grasping.
Although our base policy uses only RGB images, it is still not able to grasp the glass bottles reliably, because they differ so much from the objects it observed during training.
However, after ﬁne-tuning with only 1 hour (100 grasp attempts) of experience, we observe that the transparent bottles can be picked up with a success rate of 66%, 20% better than the base policy.
Figure 4 shows how the robot’s view changes for each challenge task.
Note the extreme glare and robot reﬂections visible in images from the Harsh Lighting challenge.
For videos of our experimental results, see the project website.
a) Collect datasets: First, we collect a dataset ofgrasp attempts for each of our 5 challenge tasks (see Table III) plus the base grasping task.
We then partitioned each dataset into 6 tiers of difﬁculty by number of exploration grasps (25, 50, 100, 200, 400, and 800 grasp attempts), yieldingindividual datasets.
b) Train ﬁne-tuned policies: We train a ﬁne-tuned policy for each of these 36 datasets using the procedure described above.
We execute the ﬁne-tuning algorithm for 500, gradient steps (see Appendix C for more information on how we chose this number) and use a learning rate of 10 − 4 , which is 25% of learning rate used for pre-training.
This yieldsﬁne-tuned policies, each trained with a different combination of target task and target dataset size.
This set of 36 policies includes 6 policies ﬁne-tuned on data from the base grasping task, for validation.
c) Train comparisons: To provide points of comparison, we train two additional policies for each challenge task and the base grasping task, yielding 12 additional policies.
The ﬁrst comparison (“Scratch”) is a policy trained using the aforementioned ﬁne-tuning procedure and an 800-grasp data set, but using a randomly-initialized Q-function rather than the Q-function obtained from pre-training.
The purpose of this comparison is to help us assess the contribution of the pre-trained parameters to the ﬁne-tuning process’ performance.
The second comparison (“ImageNet”) is also trained using an identical ﬁne-tuning procedure and the 800-grasp dataset, but uses a modiﬁed Q-function architecture in which we replace the convolutional trunk of the network with that of the popular ResNet50 architecture [21], initialized with the weights obtained by training the network to classify images from the ImageNet dataset [9].
Refer to to Fig.
9 for a diagram of the unmodiﬁed architecture.
We initialize the remaining fully-connected layers with random parameters, and concatenate the action input features at the end of the CNN (rather than the adding them in middle of the CNN, as in the original architecture).
Note that in this comparison, the ﬁnetuning process still updates all parameters, including those of the ResNet50 sub-network.
The purpose of this comparison is to provide a comparison to a strong alternative to end-to-end RL for obtaining pre-training parameters.
d) Evaluate performance: Finally, we evaluate allpolicies on their target task by deploying them to the robot and executing 50 or more grasp attempts to calculate the policy’s ﬁnal performance.
To reduce the variance of our evaluation statistics, we shufﬂe the contents of the bin between each trial by executing a randomly-generated sequence of sweeping movements with the end-effector.
The full experiment required more than 15,000 grasp attempts and 14 days of real robot time, and was conducted over approximately one month.
We present a full summary of our results in Table I.
Across the board, we observe substantial beneﬁts arising from ﬁnetuning, suggesting that the robot can indeed adapt to drastically new condition with a modest amount of data: our most dataintensive experiment uses just 0.2% of the data used train the base grasping policy to similar performance.
Our method consistently outperforms both the “ImageNet” and “Scratch” comparison methods.
We provide more detailed analysis of this experiment in the next section.
The experiments are very challenging.
For example, the “Transparent Bottles” task presents a major challenge to most grasping systems: the transparent bottles generally confuse depth-based sensors and, especially in cluttered bins, require the robot to singulate individual items and position the gripper in the right orientation for grasping.
Although our base policy uses only RGB images, it is still not able to grasp the transparent bottles reliably, because they differ so much from [Figure] (Pre- [Figure]
[Figure] QT-Opt ℒ Harsh Light Q-function Initialize
QT
[Figure]
Opt
GraspingQ-function ℒ
QT-Opt ℒ
BottlesQ-function
QT
[Figure]
Opt ℒ
ExtendQ
[Figure]
Opt
[Figure]
QT
[Figure] Transparent Bottles Checkerboard Backing [Figure]
the objects it observed during training.
However, after ﬁnetuning with only 1 hour (100 grasp attempts) of experience, we observe that the transparent bottles can be picked up with a success rate of 66%, 20% better than the base policy.
Similarly, the “Checkerboard Backing” challenge task asks the robot to differentiate edges associated with real objects from edges on an adversarial checkerboard pattern.
It never needed this capability to succeed during pre-training, where the background is always featureless and grey, and all edges can be assumed to be associated with a graspable object.
After 1 hour (100 grasp attempts) of experience, using our method the robot can grasp objects on the checkerboard background with a 71% success rate, 21% better than the base policy, and this success rate reaches 90% after 8 hours of experience ( grasp attempts).
C. Evaluating Ofﬂine Fine-Tuning for Continual Learning
Now that we have deﬁned and evaluated a simple method for ofﬂine ﬁne-tuning, we evaluate its suitability for use in continual learning, which could allow us to achieve the goal of an robot which adapts to ever-changing environments and tasks.
To do so, we deﬁne a simple continual learning challenge as follows (Fig. 3).
As in the ﬁne-tuning experiments, we begin with a base policy pre-trained for general object grasping.
Likewise, we also use our ﬁne-tuning method to adapt the base policy to a target task, in this case “Harsh Lighting.”
Not content to stop there, we use this adapted policy— not the base policy— as the initialization for another iteration of our ﬁne-tuning algorithm, this time targeting “Transparent Bottles.”
We repeat this process until we have run out of new tasks, ending at the task “Offset Gripper 10cm,” at which point we evaluate the policy on the last task.
We perform this experiment using 800 exploration-grasp datasets for each Challenge Task from our ablation study of online ﬁne-tuning with real robots.
We summarize the results in Table II.
Note that because it is the ﬁrst step of the continual learning experiment, the policy for “Harsh Lighting” is identical to that of the 800-grasp variant of the single-step experiment.
Recall that our goal for this experiment is to determine whether continual ﬁne-tuning incurs a signiﬁcant performance penalty compared to the single-step variant, because we are
[Table]
∆
[Table]
interested in using this method as a building block for continual learning algorithms.
We ﬁnd that continual ﬁne-tuning does not impose a drastic performance penalty compared to single-step ﬁne-tuning.
The continual ﬁne-tuning policies for the “Checkerboard Backing,” “Extend Gripper 1 cm ,” and “Offset Gripper 10 cm ,” challenges succeeded in grasping between 4% and 7% less often than their single-step ﬁnetuning counterparts, whereas the policy for the challenging “Transparent Bottles” case actually succeeded 8% more often.
These small deltas are within the margin-of-error of our evaluation procedure, so we conclude that the effect of continual ﬁne-tuning on the performance compared to single-step ﬁnetuning is very small.
This experiment demonstrates that our method can perform continual adaptation, and may serve as the basis for a continual end-to-end robot learning method.
IV. C ONCLUSION
For robots to be able to operate in unconstrained environments, they must be able to continuously adapt to new situations.
Our large-scale study shows that combining offpolicy RL with a very simple ﬁne-tuning procedure is an effective adaptation method, and this method is capable of achieving remarkable improvements in robot performance on new tasks with very little new data.
Furthermore, our continual learning experiment shows that using this simple method in a continual setting imposes very little performance penalty compared to the single-step setting.
This suggests that the combination of off-policy RL and ﬁne-tuning can serve as a building block for future continual learning methods.
Our results comparing supervised-learning-based initialization to those acquired with our RL-ﬁne-tuning approach highlight a familiar truism about robotics: that robotic agents must do more than perceive the world, they must also act in it.
The ability to learn the combination of these two capabilities is what makes RL well-suited for creating continually-learning robots.
A CKNOWLEDGMENTS
The authors thank Noah Brown and Ivonne Fajardo for their superb and unyielding support with real robot experiments.
We also thank Alex Irpan and Eric Jang for their help with robot learning software, Yevgen Chebotar for his advice on early revisions of this work and always-insightful discussions,
Dmitry Kalashnikov and Jake Varley for their help with QTOpt, and K.R. Zentner for her help with editing and artwork for this paper.
A PPENDIX
A [Equation] OBUSTNESS OF L EARNED P OLICIES : A C ASE S TUDY
To study the problem of adaptation, we utilize a grasping policy pre-trained with RL, which we evaluate in ﬁve different conditions that were not encountered during pre-training.
In this section, we will describe the pre-training process and test the robustness of the pre-trained policy to various robot and environment modiﬁcations.
We choose these modiﬁcations to reﬂect changes we believe a learning robot would experience, and should be expected to a adapt to, when deployed “on the job” in the real world.
In Section III, we will describe a simple ﬁne-tuning based adaptation process, and evaluate it using these modiﬁcations.
[Table]
Changing the background lighting, morphology, and objects leads to substantial degradation in performance compared to the original training conditions.
A. Pre-training process
We pre-train the grasping policy, which we refer to as the “base policy,” using the QT-Opt algorithm in two stages, as described in [29].
First, we train a Q-function network ofﬂine using data from 580,000 real grasp attempts over a corpus of 1,000 visually and physically diverse objects.
Second, we continue training this network online 3 over the course of 28,000 real grasp attempts on the same corpus of objects.
That is, we use a real robot to collect trials using the current network, update the network using these new trials, deploy the updated network to the real robot, and repeat.
This procedure yields a ﬁnal base policy that achieves 96% accuracy on a set of previously-unseen test objects.
We use a challenging subset of six of these test objects for most experiments in this work.
On this set, our base model achieves a success rate of 86% on the baseline grasping task .
B. Robustness of the pre-trained policy
We begin by choosing set of signiﬁcant modiﬁcations to the robot and environment, which we believe are characteristic of a real-world continual learning scenario.
We then evaluate the performance of the base policy on increasingly-severe versions of these modiﬁcations.
This process allows us to assess the limits of robustness of policies trained using the pre-training method.
Once we ﬁnd a modiﬁcation that is sufﬁciently-severe to compromise the base policy’s performance in each category, we use it to deﬁne a “Challenge Task” for our study of adaptation methods.
Next, we describe these challenges and the corresponding performance of the base policy.
Background: We introduce a black-white 1 inch checkerboard pattern that we glue to the bottom of the robot’s workspace (see Fig.
1, fourth from left).
We observe that conventional variations in the workspace surface, such as uniform changes in color or specularity, have no effect on the base policy’s performance.
Introducing an checkerboard pattern often fools the robot into grasping at checkerboard edges rather than objects.
This adversarial modiﬁcation compromises the base policy’s performance to 50% (-36% compared to the base task).
[Figure]
Lighting conditions: We introduce a high-intensity halogen light source parallel with the workspace (see Fig. 1, second from left), creating a bright spot in the robot’s camera view, and intense light-dark contrasts along the plane of the workspace.
The base policy was trained in standard indoor lighting conditions, with no exposure to natural light or signiﬁcant variation.
We observe that mild perturbations in lighting conditions (i.e. those which can be created by standard-intensity household lights) have no effect on the base policy’s performance.
Using the very bright halogen light source has a severe impact, and degrades the base policy’s performance to 31% (-55% compared to the baseline).
Gripper shape: We extend the parallel gripper attached to the robot by 1 cm and signiﬁcantly narrow its width and compliance in the process (see Fig.
1, ﬁfth from left).
This changes the robot’s kinematics (lengthening the gripper in the distal direction), while also lowering the relative pose of the robot with respect to the workspace surface by 1 cm .
This modiﬁcation compromises the base policy’s performance to 76% (-10% compared to the baseline).
Robot morphology:
We translate the gripper laterally by 10 cm (see Fig.
1, far-right).
Note that during training this policy experienced absolutely no variation in robot morphology.
We observe that translating the gripper laterally by up to 5 cm has no impact on performance.
By translating the gripper laterally by 10 cm (approximately a full gripper or arm link width), we degrade the base policy’s performance to 47% (- 39% compared to the baseline).
Unseen objects: We introduce completely-transparent plastic beverage bottles (see Fig.
1, third from left) that were not present in the training set.
Based on our experiments, the system is robust to a broad variety of previously-unseen objects, as long as they have signiﬁcant opaque components.
For example, even though there are no drinking bottles in the training set, we ﬁnd the system is able to pick up labeled drink bottles with 98% success rate.
Success rates for other novel, opaque objects are similarly consistent with the baseline performance on the test set.
However, we ﬁnd that introducing completely-transparent drink bottles causes the base policy to often grasp where two bottles are adjacent, i .
e .
as though it cannot differentiate which parts of the scene are inside vs outside a bottle.
By introducing completely-transparent plastic beverage bottles, we are able to compromise the base policy’s performance to 49% (-37% compared to the baseline).
See Table III for a summary of the modiﬁcation experiments, and their effect on base policy performance.
A PPENDIX B A SSESSING F INE -T UNING T ECHNIQUES FOR E ND - TO -E ND R OBOTIC R EINFORCEMENT L EARNING
We propose a conceptual framework for ﬁne-tuning algorithms, and use simulation experiments to assess the suitability of some algorithm variations for end-to-end robot learning.
“Fine-tuning” refers to a family of transfer learning techniques, in which we seek to acquire a neural network for one task (which we will refer to as the “target” task) by making use of some or all of a network trained on a related task (the “base” task).
This is a is a very common technique for quickly acquiring new tasks in computer vision [11, 25, 34] and natural language processing [23].
As collecting new robot experience data is expensive, our goal is to use as little target task data as possible.
In this section, we ﬁrst describe the general algorithmic sketch for ﬁne-tuning, then enumerate some of the most common ﬁne-tuning techniques.
In Sections III and C, we evaluate the suitability of these techniques for end-to-end robot learning.
A. Fine-Tuning: Conceptual Framework
We can organize ﬁne-tuning for end-to-end reinforcement learning into four essential steps (Fig. 2).
Different ﬁne-tuning techniques change the details of one of these steps.
1) Pre-training : Pre-train a policy to perform some base task, which is related to our target task. In the experiments
in
this
work , the base task is always indiscriminate object grasping.
In computer vision and NLP, this step can often by skipped by making use of one of many pre-trained and publicly-available state-of-the-art vision and language models.
We hope for a future in which this is possible in robotics.
2) Exploration : Explore in the new target task, to collect data for adaptation.
In principle, in off-policy reinforcement learning any policy may be used for exploration.
In our study, and what we believe to be most representative of a real-world continual learning scenario, we always use the pre-trained policy for exploration.
3) Initialization : Initialize the policy for the target task using some or all of the weights from the pre-trained policy.
The standard implementation of this step is to start with the entire pre-trained network.
Some techniques may choose to use only a subset of the pre-trained network ( e .
g .
truncating the last few layers of a CNN).
4) Adaptation : Use the exploration data update the initialized policy to perform the new task.
The standard version of this step continues updating the entire initialized policy with the same algorithm and hyperparameters [Figure] has a negative effect on sample efﬁciency for ﬁne-tuning. as was used for the pre-training process, but with the target task data.
There are many variations on this step, including which parts of the network to update, at what learning rate, with what data, with which optimization algorithm, whether to add additional network layers, etc.
5) Evaluation : Assess performance of the ﬁne-tuned network on the new task.
If this step only happens once, we refer to such a technique as “ofﬂine ﬁne-tuning,” because the adaptation step never uses data from an updated policy.
If this step happens repeatedly ( e . g .
exploration and evaluation are one-and-the-same), and its result is used for further adaptation to the same target task, we refer to a technique as “online ﬁne-tuning.”
We explore both variations in our experiments.
Using this ﬁne-tuning framework, we consider several variations of ﬁne-tuning, and assess their suitability for end-to-end robotic RL.
Notably, we neglect an analysis of pre-training techniques for ﬁne-tuning reinforcement learning (i.e. (1)), which has a large and rapidly-growing body of research in the meta- and multi-task RL communities (see Sec. II).
Instead, we focus on initialization (2) and adaptation (3).
All of our experiments use end-to-end off-policy reinforcement learning of an indiscriminate object grasping task for their pre-training step.
Refer to Section A-A for details on our pre-training process.
B. Experiments in simulation
We use simulation experiments to evaluate the suitability of some ﬁne-tuning variations, along the axes we deﬁned in
Section B-A.
1) Adding a new head and other selective initialization techniques: Selective-initialization techniques start the ﬁnetuning process with a policy which has some of its parameters initialized to random, e .
g .
a popular variant is to “add a head” to a pre-trained neural network by omitting its last few layer(s) [Figure]
[Figure]
from initialization, so that the new head can be trained to perform on the target task.
Figure 5 portrays a study of partial initialization for online ﬁne-tuning using a simulated grasping experiment.
In this experiment, the base task is “grasp opaque blocks” and the target task “grasp semi-transparent blocks,” and the base policy performance is 98% when trained from scratch on 43,000 grasp attempts.
Both ﬁne-tuned policies begin with low performance, around 15%.
After 5000 exploration grasps (12% of the data used for the base policy), the performance of the full initialization policy has reached the base policy performance, while the policy with a new head has barely reached 30%.
This gap shows that the combination of offpolicy RL and selective initialization is unsuitable for sampleefﬁcient ﬁne-tuning.
Our experiments immediately make apparent the downsides of selective initialization for ﬁne-tuning.
In particular, online ﬁne-tuning requires to maintain a policy that can competently explore the target task at all times, any method which compromises the performance of such a policy–even temporarily–has a high risk of failing as a sample-efﬁcient ﬁne-tuning technique.
The resulting performance gap, once created, is hard to recover from.
As a consequence, we ﬁnd in simulation experiments that online ﬁne-tuning with selective re-initialization takes a signiﬁcant fraction of the pre-training samples to converge to baseline performance, making this family of ﬁne-tuning methods sample inefﬁcient.
C. Training with a mix of data from the base and target tasks
We experiment with mixing data from the pre-training task into the ﬁne-tuning process (Fig. 6), and ﬁnd that in simulation this has a predictable relationship with sample efﬁciency:
[Figure]
higher shares of target task data allow the ﬁne-tuning policy achieve higher performance faster.
Our goal is to design a ﬁne-tuning algorithm for real robots which might be used for continual learning, and our conclusion from this brief study is that online ﬁne-tuning is a poor ﬁt for for this goal.
The experiments with selective re-initialization in particular highlights the challenge of online ﬁne-tuning: it only allows us to use algorithms which preserve the exploration ability of the policy at all times.
We also believe that ofﬂine ﬁne-tuning is more practical than online ﬁne-tuning, due to the inherent complexity of placing a robot in the loop of a reinforcement learning algorithm.
If used as part of a continual learning method, an ofﬂine method would also allow a robot to collect data on a new task piecemeal, and only attempt to adapt to that new task when it has collected enough data to be successful.
A PPENDIX C E MPIRICAL A NALYSIS
In this section, we aim to further investigate the efﬁciency, performance, and characteristics of our large-scale real-world adaptation experiments. A. Performance and sample efﬁciency of our method Figure 7 shows the success rates for our method from Table I against the amount of data used to achieve that success rate for selected tasks.
The data indicates that our simple ofﬂine ﬁne-tuning method can adapt policies to many new tasks with performance at or even above the state-of-the-art base policy, using modest amounts of data.
For instance, “Extend
Gripper 1cm” and “Offset Gripper 10cm” both needed onlyexploration grasps to achieve substantial gains in performance (+18% and +30%, respectively).
All policies attain substantial performance gains over the base policy by the time they are exposed to 800 exploration grasps, which is less than 0.2% of the data necessary to train an equivalently-performing policy on the base task.
While the general trend is that more exploration data leads to higher performance, this relationship is not linear.
All
[Figure]
methods experience a substantial improvement in performance after 100 or fewer exploration grasps.
However, we observe that these performance improvements in the very low-data regime ( e .
g .
≤ 200 grasp attempts) are also unstable.
B. The downside of ofﬂine ﬁne-tuning: deciding when to stop
Our results indicate that ofﬂine ﬁne-tuning can train robotic policies to substantial performance improvements with modest amounts of data, and that ofﬂine methods are not limited by the need to preserve an always-sufﬁcient exploration policy as with online methods. However, we identify one signiﬁcant drawback to the method compared to online ﬁne-tuning. A pure ofﬂine ﬁne-tuning method has no built-in evaluation step which would inform us when the robot’s performance on the target task has stopped improving, and therefore when we should stop ﬁne-tuning with a ﬁxed set of target task data. This is a subset of the off-policy evaluation problem [26].
Knowing when the policy stops improving is important, because ﬁnetuning exists in a low-data regime, and repeatedly updating a neural network model with small amounts of data leads to overﬁtting onto that data.
Not only does this degrade the performance on the target task, but also the ability of the network to adapt to new tasks later ( i .
e .
for continual learning).
We can see this phenomenon in Figure 8 showing a real robot’s performance on the “Offset Gripper 10cm” target task at different numbers of steps into an ofﬂine ﬁne-tuning process that uses 400 exploration grasps.
Performance quickly rises until around 500,000 gradient steps.
Past this point, it precipitously drops and never recovers, dropping below even the initial performance of the base policy from which it was trained, as the initialization is being overwritten by overﬁtting to the target samples.
The point at which overﬁtting begins is a function of the initialized model, target dataset, learning algorithm, and many other factors, and is not necessarily stable or easily predictable.
For the purposes of our large-scale ﬁne-tuning study, we use this experiment and several others to determine that 500, gradient steps was an acceptable choice for the real-world experiments, but the variance in the results in Table I and
Figure 7 shows that this choice was not necessarily optimal for all of our tasks and datasets.
We believe one practical a solution to this problem of a continual learning robot is to use a mix of ofﬂine ﬁne-tuning and online evaluation.
The point, at which performance stops improving represents when the training process has exhausted the ﬁne-tuning dataset of new information, and the robot must return to exploring online to continue improving.
C. Comparing initializing with RL to initializing with supervised learning
In order to answer the question whether RL is better suited for creating a continually-learning robotic agent than supervised learning, we compare our results to an ImageNetpretrained baseline.
The ImageNet baseline uses a similar grasping network where its convolutional layers are replaced with ResNet 50 architecture and pre-loaded with ImageNet features.
Since the part of the network that process robot’s state and action inputs cannot be initialized using supervised learning, we initialize them randomly.
As shown in Table I, the best performing ImageNet-based agent achieves the success rate of 47% on “Offset Gripper 10cm,” which corresponds to 4% improvement over the base policy performance.
This result seems to conﬁrm our hypothesis that our RL-based pretraining is crucial for good subsequent ﬁne-tuning.
Note that we ﬁrst attempted to ﬁne-tune these ImageNet-based policies while holding the ImageNet feature layers constant, but this procedure failed to achieve any non-zero success rate.
This suggests that, unlike adapting computer vision networks to new visual tasks, adapting end-to-end robot learning to new sensorimotor tasks may require changing the features used to represent the problem, and not just the post-processing of said features.
Figure 9 highlights some of the changes that happen during the RL-based ﬁne-tuning in greater detail.
It demonstrates the normalized distance in parameter space of a ﬁne-tuned policy for each of our challenge tasks from its base policy.
While it is unsurprising that primarily-visual challenges such as “Checkerboard Backing” and “Harsh Lighting” induce large changes in the parameters of the convolutional parts of the network, we observe that even ‘Offset Gripper 10cm,” a purely-morphological change to the robot, induces substantial changes to the network’s image-processing parameters (e.g. layers conv2-conv7).
We attribute this to the successful agent’s need for hand-eye coordination to complete the task: offsetting the gripper not only changes robot morphology, it changes the location of the robot in its own visual ﬁeld drastically.
In order c o n vc o n vc o n vc o n vc o n vc o n vc o n vc o n vc o n vc o n vc o n vc o n vc o n vc o n vc o n vf c 0 f c 1 l
o
g
i t p
i x e l s
C h
e c k e r b o a r d B a c k
i n g
H a r s h L i g h t
i n g
O ﬀ s e t G r i p p
e r 1 0 c m
E x t e n d
G r i p p e r 1 c m T r a n s p a r e n t B o tt l e s c o n vf [Figure]
e x p . r e t u r n to perform effective visual servoing with a new morphology, both the image and action-processing parts of the network must be updated. A PPENDIX D A DDITIONAL R ELATED W ORK Reinforcement learning is a long-standing approach for enabling robots to autonomously acquire skills [32] such as locomotion [33, 64], pushing objects [37, 12], ball-in-cup manipulation [31], peg insertion [16, 36, 59, 35, 71], throwing objects [15, 72], and grasping [49, 29].
We particularly focus on the problem of deep reinforcement learning from raw pixel observations [36], as it allows us to place little restrictions on state representation.
A number of works have also considered this problem setting [13, 15, 12, 71, 1, 40].
However, a key challenge with deep RL methods is that they typically learn each skill from scratch, disregarding previously-learned skills.
If we hope for robots to generalize to a broad range of real world environments, this approach is not practical.
Other works have aimed to share and transfer knowledge across tasks and conditions by simultaneously learning across multiple goals and tasks [54].
For example, prior works in model-based RL [12, 67, 43] and in goal-conditioned RL [1, 44, 47, 50, 70] have shared data and representations across multiple goals and objects.
Along a similar vein, prior work in robotic meta-learning has aimed to learn representations that can be quickly adapted to new dynamics [41, 2, 42] and objects [14, 27, 69, 3].
We consider adaptation to a broad class of changes including dynamics, object classes, and visual observations, including conditions that shift substantially from the training conditions, and do not require the full set of conditions to be represented during the initial training phase.