Scaling Up Multi-Task Robotic Reinforcement Learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract: General-purpose robotic systems must master a large repertoire of diverse skills.
While reinforcement learning provides a powerful framework foracquiring individual behaviors, the time needed to acquire each skill makes theprospect of a generalist robot trained with RL daunting.
We study how a largescale collective robotic learning system can acquire a repertoire of behaviors simultaneously, sharing exploration, experience, and representations across tasks.
In this framework, new tasks can be continuously instantiated from previouslylearned tasks improving overall performance and capabilities of the system.
Tothis end, we develop a scalable and intuitive framework for specifying new tasksthrough user-provided examples of desired outcomes, devise a multi-robot collective learning system for data collection that simultaneously collects experiencefor multiple tasks, and develop a scalable and generalizable multi-task deep reinforcement learning method, which we call MT-Opt.
We demonstrate how MT-Optcan learn a wide range of skills, including semantic picking (i.e., picking an objectfrom a particular category), placing into various ﬁxtures (e.g., placing a food itemonto a plate).
We train and evaluate our system on a set of 12 real-world tasks withdata collected from 7 robots, and demonstrate the performance of our system bothin terms of its ability to generalize to structurally similar new tasks, and acquiredistinct new tasks more quickly by leveraging past experience.
We recommendviewing the videos at https://mt-opt.github.io/
Introduction
Today’s deep reinforcement learning (RL) methods, when applied to real-world robotic tasks, provide an effective but expensive way of learning skills [1, 2].
While existing methods are effective andable to generalize, they require considerable on-robot training time.
For example, the QT-Opt [1]system can learn vision-based robotic grasping, but it requires over 500 , 000 trials collected acrossmultiple robots.
While such sample complexity may be reasonable if the robot needs to perform asingle task, it becomes costly if we consider the prospect of training a general-purpose robot with alarge repertoire of behaviors, where each behavior is learned in isolation, starting from scratch.
Canwe instead amortize the cost of learning this repertoire over multiple skills, where the effort neededto learn whole repertoire is reduced compared to learning each skill in isolation?
Prior work suggests that multi-task RL can amortize the cost of single-task learning [3, 4, 5, 6, 7].
Insofar as the tasks share common structure, if that structure is discovered by the learning algorithm,all of the tasks can in principle be learned more efﬁciently than learning each task individually.
In addition, by collecting experience simultaneously using controllers for a variety of tasks withdifferent difﬁculty, the easier tasks can “bootstrap” the harder tasks.
Finally, by enabling the multitask RL policy to learn shared representations, learning new tasks can become easier over time asthe system acquires more skills and learns more widely-useful aspects of the environment.
However, to realize these beneﬁts for a real-world robotic learning system, we need to overcome anumber of major challenges [8, 9, 10, 11], which have so far made it difﬁcult to produce a largescale demonstration of multi-task image-based RL in the real world.
First, multi-task reinforcement learning is known to be exceedingly difﬁcult from the optimization standpoint [8, 12].
Second, areal-world multi-task learning framework requires the ability to easily and intuitively deﬁne rewardsfor a large number of tasks.
Third, while all task-speciﬁc data could be shared between all thetasks, it has been shown that reusing data from non-correlated tasks can be harmful to the learningprocess [13].
Lastly, to receive the beneﬁts from shared, multi-task representation, we need tosigniﬁcantly scale up our algorithms, the number of tasks, and the robotic systems themselves.
The main contribution of this paper is a generalmulti-task learning system, which we call MT-Opt.
MT-Opt realizes the hypothesized beneﬁts of multitask RL in a scalable real-world robotic learning system, addressing the associated challenges througha number of important design decisions, includinga carefully designed multi-task data sharing strategy and multi-robot parallelized data collection.
Wetrain MT-Opt on a set of 12 real-world tasks thatinclude data collected over the course of over 1.5years, and provide an extensive evaluation (spanning over 500 real-robot-hours) of our method together with a number of baselines.
We show that atsuch scale, our system can quickly acquire new tasksby taking advantage of prior tasks via shared representations, new data-sharing strategies and learnedpolicies.
Our results indicate that, by learning multiple related tasks simultaneously, not only can we increase the data-efﬁciency of learning each of them,but also solve more complex tasks than in a singletask setup.
We present our multi-task system as wellas examples of some of the tasks that it is capable ofperforming in Fig.
Related Work
Multi-task learning, inspired by the ability ofhumans to transfer knowledge between differenttasks [14], is a promising approach for sharing structure and data between tasks to improve overall efﬁciency.
Multi-task architectures have been successful across multiple domains; natural languageprocessing [15, 16, 17, 18] and computer vision [19, 20, 21, 22, 23, 24].
In this work, we applymulti-task learning concept in a RL setting to real robotic tasks.
Combining multiple task policies has been explored in RL by using gating networks [25, 26], conditioning policies on tasks [27], mapping tasks to parameters of a policy [28, 29, 30], distilling separatetask policies into a shared multi-task policy [31, 32, 33, 34, 35, 36].
Advantages of multi-task learning for visual representations has been explored in [37].
Pinto and Gupta [4] use a shared multi-taskneural network architecture, which is trained with separate task-speciﬁc losses.
In contrast, in ourwork, we concentrate on tasks with a common loss structure within a Q-learning framework.
Several works explore how to mitigate multi-task interference and conﬂicting objectives [38, 39].
Inour experiments, we ﬁnd that better data routing helps with not only better mitigating conﬂictingobjectives but also improving learning efﬁciency through data reuse.
Learning complex skills has been addressed through hierarchical reinforcement learning with options [40, 41, 42], combining multiple sub-tasks [43, 44, 45, 46, 47, 48], reusing samples betweentasks [49], relabeling experience in hindsight [50], introducing demonstrations [51, 52, 53, 54, 55,56, 57].
A range of works employ autonomous supervision to learn diverse skills, e.g. by scalingup data collection [58], sampling suitable tasks [59] or goals [60] to practice, learning a task embedding space amenable to sampling [7], or learning a dynamics model and using model-predictivecontrol to achieve goals [61, 62, 63, 64].
Riedmiller et al.
[5] learn sparse-reward tasks by solvingeasier auxiliary tasks and reusing that experience for ofﬂine learning of more complex tasks.
In Cabiet al.
[65], previously collected experience is relabeled with new reward functions in order to solvenew tasks without re-collecting the data.
In our work, we similarly design techniques for reusingexperience between related tasks and apply them to large-scale data collection on real robots.
MT-Opt: a Scalable Multi-Task RL System
[Figure]
Fig.
2 overviews our multi-task learning sys-
[Figure] tem.
We devise a distributed, off-policy multi-
[Figure] task RL algorithm together with a multi-task [Figure] visual success detector ( SD ) to learn multiple robotic manipulation tasks simultaneously.
The SD , trained from video examples of desired outcomes, (Fig.
2A) in part determineshow episodes are leveraged to train an RL policy (Fig.
2C).
During evaluation and ﬁne-tuning(Fig.
2B), at each time step, a policy takes asinput a camera image and a one-hot encodingof the task, and sends a motor command to therobot.
At episode end, the outcome image ofthis process is graded by the SD that determines which tasks were accomplished successfully and assigns a sparse reward 0 or 1 foreach task.
The system then decides whether another task should be attempted or if the environment should be reset.
We train this system withmultiple robots, where each robot concurrentlycollects data for a different, randomly selectedtask.
The generated episodes are used as ofﬂinedata for training future policies (Fig.
2C) andare available to improve SD s.
In order to enable our system to take full advantage of multi-task training, we develop a number of components and evaluate their importance for overall performance.
First, we use asingle, multi-task deep neural network to learna policy for all the tasks simultaneously, whichenables parameter sharing between tasks.
Second, we devise data management strategies that shareand re-balance data across certain tasks.
Third, since all tasks share data and parameters, we usesome tasks as exploration policies for others, which aids in exploration.
Although the individualcomponents are based on previously studied concepts in multi-task learning, we show through ourexperiments that the particular choice and design of these components is essential for our largescale multi-task learning system to effectively leverage data from all tasks to improve performance,particularly for “underrepresented” tasks that have the least available training data.
3.1 Multi-Task Reinforcement Learning Algorithm
We denote the multi-task RL policy as π ( a | s , T i ) , where a ∈ A denotes the action, which in ourcase is the desired change in end-effector position, a change in yaw angle, binary gripper open andclose commands and a termination command, s ∈ S denotes the state, which corresponds to imagesfrom the robot’s cameras, and T i denotes an encoding of the i th task drawn from a categorical taskdistribution T i ∼ p ( T ) , with n possible categories, each corresponding to a different task.
At eachtime step, the policy selects an action a given the current state s and the current task T i set at thebeginning of the episode, and receives a task-dependent reward r i ( a , s , T i ) .
The goal of the multitask RL policy is to maximize the expected sum of rewards for all tasks drawn from the distributionp ( T ) .
The episode ﬁnishes when the policy selects a TERMINATE action or reaches a step limit.
We build on the single-task QT-Opt algorithm [1], which itself is a variant of Q-learning [66],and learns a single-task optimal Q-Function by minimizing the Bellman error:
[Equation]
) = 148 E ( s , a , s (cid:48) ) ∼ p ( s , a , s (cid:48) ) (cid:2) D ( Q θ ( s , a ) , Q T ( s , a , s (cid:48) )) (cid:3) , where Q T ( s , a , s (cid:48) )
=
r ( s , a ) + γV ( s (cid:48) ) is a target
Q-value and D is a divergence metric, such as cross-entropy, γ is a discount factor, V ( s (cid:48) )
isthe target value function of the next state computed using stochastic optimization of the form
V ( s (cid:48) )
= max a (cid:48) Q ( s (cid:48) , a (cid:48) ) , and the expectation is taken w.r.t.
previously seen transitions p ( s , a , s (cid:48) ) .
Similarly to [1], we use the cross-entropy method (CEM) to perform the stochastic optimization tocompute the target value function.
To extend this approach to the multi-task setting, let s ( i ) , a ( i ) , s (cid:48)
( i ) denote a transition that wasgenerated by an episode e ( i ) for the i th task T i .
Each transition could be used for multiple tasks.
Inthe multi-task case, the loss becomes (with ( s ( i ) , a ( i ) , s (cid:48) ( i ) ) being transitions generated by tasks T i ):
[Equation] [Equation]
[Figure] [Equation] [Figure] [Equation]
[Figure]
3.2 Sharing Data Between Tasks
S F
S F S F
S F S F
S F S F
S F
S F . . . . .
procedure f I ( e i : original episode) expanded episodes =
[] SD { k i } ← set of SDs relevant to task T
i for SD k in SD { k i } do //
e k : e i but rewards for task
T k not T i e k
= SD k ( e i ) expanded episodes.append( e k ) return expanded episodes One advantage of an off-policy RL algorithm isthat collected experience can be used to updatethe policy for other tasks, not just the task forwhich it was originally collected.
This sectiondescribes how we effectively train with multitask data through task impersonation and data rebalancing , as summarized in Fig.
We leverage experience sharing at the episodelevel rather than the individual transition level.
The goal is to use all transitions of an episodee ( i ) generated by task T i to aid in training a policy for a set of k i tasks T { k i } .
We refer to thisprocess as task impersonation (see Algorithm 1),where the impersonation function f I transformsepisode data collected for one task into a set ofepisodes that can be used to also train other tasks,i.e.: e { k i } = f I ( e ( i ) ) .
Note that in general case{ k i } is a subset of all tasks { n } , and it dependson the original task T i that the episode e ( i ) wascollected for.
We consider two ends of the data-sharingspectrum: an identity impersonation functionf I orig ( e ( i ) )
=
e ( i ) , where no task impersonationtakes place, i.e. an episode e ( i ) generated by task
T i is used to train the policy only for that task; and f I all = e { n } , where each task shares data with alln − 1 tasks (maximal data sharing).
While f I orig fails to leverage the reusable nature of multi-taskdata, f I all results in many unrelated episodes used as negative examples for the target task.
For MT-Opt, we devise a new task impersonation strategy
f I skill that uses more ﬁne-grained similarities between tasks.
We refer to it as a skill-based task-impersonation strategy, where we overload theterm “skill” as a set of tasks that share semantics and dynamics, yet can start from different initialconditions or operate on different objects.
In this work we manually assign each task to a particularskill based on the subjective similarity of the required manipulation primitive.
For example, taskssuch as place-object-on-plate and place-object-in-bowl belong to the same skill.
Our impersonationfunction f I skill allows us to impersonate an episode e ( i ) only as the tasks belonging to the same skillas T i . Namely, given an episode e i generated by task T i , a skill S j that task belongs to is detected. The e i will be impersonated only for the tasks T { S j } belonging to that particular skill. In order toavoid impersonating too many episodes and overloading the replay buffer, we introduce a stochastic impersonation function.
An impersonated episode candidate will be routed to training with theprobability p s if it’s a success, or with probability p f if it’s a failure. We provide more details on thespeciﬁc parameters in the Appendix 7.3.
In our experiments, we conduct ablation studies comparingf I skill (ours) with other task impersonation strategies.
While training, due to the design of our task impersonation mechanism, as well as the variabilityin difﬁculty between tasks, the resulting training data stream often becomes highly imbalanced, see
Fig.
3B. To address this, we re-balance each batch both between tasks, such that the relative proportion of training data for each task is equal, and within each task, such that the relative proportion ofsuccessful and unsuccessful examples is kept constant.
SD -Based Rewards and Continuous Data Collection
We learn tasks evaluated on the ﬁnal state of an episode.
This sparse-reward assumption allows us totrain a multi-view multi-task CNN success detector model ( SD ),conditioned on task ID, which given3 ﬁnal images from 3 different camera angles, infers task success (see Appendix 7.1 for details).
To generate training data for the SD , we ask users to collect positive and negative examples of positive and negative outcomes of a task.
These examples are not demonstrations – just examples ofwhat successful completion (i.e., the ﬁnal state) looks like.
We use this data to train the initial version of the multi-task SD , which we then continuously ﬁne-tune given the images gathered duringautonomous data collection to avoid out of distribution images that might be caused by various realworld factors such as different lighting conditions, and novel states which the robot discovers.
Wecontinue to manually label such images and incrementally retrain SD to obtain the most up-to-date
SD .
In result, we label ≈ 5 , 000 images per task and provide more details in the Appendix 8.
[Figure]
Our main observation w.r.t.
the multi-task datacollection is the use of easier tasks to bootstrap learning of more complex tasks.
In particular, an average MT-Opt policy for simpletasks might occasionally yield episodes successful for harder tasks.
Over time, this allowsus to start training an MT-Opt policy now forthe harder tasks, and consequently, to collectbetter data for those tasks.
To kick-start thisprocess and bootstrap our two simplest tasks,we use two crude scripted policies for pickingand placing (see Sec.
7.2 for details) following prior work [1].
To simplify exploration forlonger-horizon tasks, we allow the individualtasks to be ordered sequentially, where one taskis executed after another using simple heuristics such as executing placing tasks after liftingtasks.
Our dataset grows over time w.r.t.
theamount of per-task data as well as percentageof successful episodes for all the tasks.
All datafrom a variety of different policies is accumulated over time: different versions of trained policies,their epsilon-greedy counterparts ( (cid:15) = 20% ), and scripted policies.
Importantly, this data collection process results in an imbalanced dataset, as shown on Fig.
Our data impersonation andre-balancing methods address this imbalance by efﬁciently expanding and normalizing data.
Experiments
The goal of our real-world experiments is to answer the following questions: (1) How does MT-Optperform, quantitatively and qualitatively, on a large set of vision-based robotic manipulation tasks?(2) Does training a shared model on many tasks improve MT-Opt’s performance?
(3) Does datasharing improve performance of the system?
(4) Can we use easier tasks to bootstrap learning ofmore difﬁcult tasks?
(5) Can MT-Opt quickly learn distinct new tasks by adapting learned skills?
5.1 Experimental Setup
MT-Opt provides a general robotic skill learning framework that we use to learn multiple tasks,including semantic picking (i.e., picking an object from a particular category), placing into various ﬁxtures (e.g., placing a food item onto a plate).
We focus on basic manipulation tasks that requirerepositioning objects relative to each other.
A wide range of manipulation behaviors fall into thiscategory, from simple bin-picking to more complex behaviors, such as covering items with a cloth. As described in Sec. 4, our task set grows over time allowing for more and more tasks to be similar.
To analyze and ablate various properties of MT-Opt, we focus on a set of different but similar tasks,where multi-task learning shows maximal effectiveness.
In the following experiments, we use aset of 12 tasks for quantitative evaluation of our algorithm.
These 12 tasks include a set of plasticfood objects and divided plate ﬁxtures and they can be split into “object acquisition” and “objectmanipulation” skills.
Since the criteria of what constitutes a task are not well deﬁned, in our system,different tasks correspond to different success conditions as deﬁned by our SD .
Our most generalobject acquisition task is lift-any , where the goal is to singulate and lift any object.
We also deﬁne7 semantic lifting tasks, where the goal is to search for and lift a particular object (ex: carrot).
Importantly, these semantic tasks require search in clutter and singulation before the actual pickingcan be performed.
The placing tasks utilize a divided plate where the simplest task is to place thepreviously lifted object anywhere on the plate ( place-any ).
Harder tasks require placing the objectinto a particular section of a divided plate, oriented arbitrarily.
Fig.
5 visualizes the tasks.
All polices are trained with ofﬂine RL from a large dataset summarized in Fig.
The resultingpolicy is deployed on 7 robots attempting each task 100 times for evaluation.
To reduce the varianceof the evaluation, we shufﬂe the bins after each episode and use a standard evaluation scene (see Appendix, Fig. 17), where all of objects are present, hence all 12 evaluation tasks are feasible.
5.2 Quantitative and Qualitative Evaluation of MT-Opt
Fig.
6 shows MT-Opt success rates on the 12 evaluation tasks.
We compare to three baselines: (i) single-task QT-Opt [1], where each per-task policy is trained separately using only data collected speciﬁcally for that task, (ii) an enhanced QT-Opt baseline, which we call QT-Opt Multi-Task,where we train a shared policy for all the tasks but there is no data impersonation or re-balancingbetween the tasks, and (iii) a Data-Sharing Multi-Task baseline that is based on the data-sharingstrategy presented in [65], where we also train a single Q-Function but the data is shared acrossall tasks.
From the average performance across all task, we observe that MT-Opt outperforms thebaselines, in some cases with ≈ 3 × average improvement.
While the single task QT-Opt baseline performs similarly to MT-Opt for the task where we have the most data (see data statisticsin Fig. 4), lift-any , its performance drastically drops (to ≈ 1% ) for underrepresented tasks, suchas lift-can .
Note, that we are not able run this baseline for the placing tasks, since they require aseparate task to lift the object, which is not present in the single-task baseline.
A similar observation applies to QT-Opt Multi-Task, where the performance of rare tasks increases compared to QT-Opt, but is still ≈ 4 × worse on average than MT-Opt.
Sharing data across all tasks also resultsin a low performance for semantic lifting and placing tasks and, additionally, it appears to harmthe performance of the indiscriminate lifting and placing tasks.
The MT-Opt policy, besides attaining the 89% success rate on ( lift-any ), also performs the 7 semantic lifting tasks and the 4 placingand rearrangement tasks at a signiﬁcantly higher success rate than all baselines.
Since all ablatedexperiments are trained on the same exact ofﬂine dataset (except for the case where the multiplepolicies are trained on the per-task shard of the dataset) and have the same computation budget,we explain these performance gaps by the way MT-Opt shares the representations and data, andprovide a more comprehensive analysis of these factors in the following experiments.
To highlightlearned behaviors, Fig.
7 shows the policy work the carrot out of the corner before picking it up.
[Table]
Fig. 4, shows amounts and type of data collected for various tasks.
Tasks such as lift-carrot and lift-bottle , whichhave more data, especially on-policy data, have higher success rates than underrepresented tasks, such as lift-box .
Theperformance of these underrepresented tasks could be furtherimproved by focusing the data collection on these tasks.
5.3 Sharing Representations Between Tasks
To explore the beneﬁts of training a single policy on multipletasks, we compare the 12-task MT-Opt policy with a 2-task policy that learns lift-any and placeany .
Both of these policies are evaluated on ( lift-any and place-any ).
We use the same f I skill taskimpersonation strategy, and the same ofﬂine dataset (i.e. both policies use the data from the extra 10 narrower tasks, which is impersonated as lift-any and place-any data) without on-policy ﬁne-tuning,so data-wise the experiments are identical.
The 12-task policy outperforms the 2-task policy even on the two tasks that the 2-task policy istrained on (Table 1), suggesting that training multiple tasks not only enables the 12-task policy toperform more tasks, but also improves its performance on the tasks through sharing representations.
This may suggest that the additional supervision provided by training on more tasks has a beneﬁcialeffect on the shared representations.
5.4 Data Sharing Between Tasks
To test the inﬂuence of data-sharing and re-balancing on the multi-task policy’s performance, wecompare our task impersonation strategy
f I skill introduced in Sec.
3.2 to a baseline impersonationfunction that does not share the data between the tasks f
I orig , and a baseline where each task isimpersonated for all other tasks f
I all .
In our skill-based task impersonation strategy f I skill , the data isexpanded only for the class of tasks having similar visuals, dynamics and goals.
We also test datare-balancing strategy, where we re-balance each training batch between the tasks as well as withineach task to keep the relative proportion of successful and unsuccessful trials the same.
[Table]
complimentary: they both improve over the baselines, while the best effect is achieved by combining both.
The effect is especially pronounced for the underrepresented tasks.
The results of this experiment arein Table 2, with the full resultsreported in the Appendix, Table 4.
Sharing data among tasksusing our method of task impersonation and re-balancing provides signiﬁcant improvementacross all the tasks, with improvements of up to 10 x forsome tasks.
The full data-sharingstrategy performs worse thanboth the no-data-sharing baselineand our method, suggesting thatna¨ıvely sharing all data across all tasks is not effective .
Because of our data-collection strategy, theresulting multi-task dataset contains much more data for broader tasks (e.g., lift-any ) than for morenarrow, harder tasks (e.g., lift-box ), as shown in Fig.
Without any additional data-sharing andre-balancing, this data imbalance causes the baseline strategy f
I orig to attain good performance forthe easier, overrepresented tasks, but poor performance on the harder, underrepresented tasks (see
Table 2, ﬁrst row), whereas our method performs substantially better on these tasks.
5.5 Using Easier Tasks to Bootstrap Harder Tasks
To explore question (4), we study whether learning an easier, broader task ( lift-any ) helps with astructurally related task that is harder, more speciﬁc ( lift-sausage ).
We separate out data for liftsausage which consists of 5400 (nearest hundred) episodes collected for that task (i.e. 4600 failuresand 800 successes).
In addition, there are 11200 episodes of successful sausage lifting and as manyas 740 K failures that were collected during the lift-any task.
Combining the lift-sausage data andthe extra successes from lift-any yields 16600 episodes ( 12000 successes and 4600 failures).
Toinvestigate the inﬂuence of MT-Opt and task impersonation on the bootstrap problem, we compareour 12-task MT-Opt policy to a single-task policy trained on these 16600 episodes, including thesame set of successful lift-sausage episodes as MT-Opt, but not including failures from other tasks.
The single-task policy learned from the 16600 episodes yields performance of 3% .
MT-Opt, usingimpersonated successes and failures, achieves 39% success for the same task, a ≈ 10 × improvement.
Both experiments use identical data representing successful episodes.
The beneﬁts of MT-Optare twofold.
First, we leverage an easier lift-any task to collect data for the harder lift-sausage task.
Second, MT-Opt beneﬁts from the additional failures impersonated from other tasks.
These failures, which often include successful grasps of non-target objects, when re-balanced as describedin Sec. 3.2, results in the signiﬁcant task performance boost.
This demonstrates the value of bothsuccessful and unsuccessful data collected by other tasks for learning new tasks.
5.6 Learning New Tasks with MT-Opt
In addition to retroactive relabelling of prior data, MT-Opt can learn new tasks via proactive adaptation of known tasks, even ones that are visually and behaviorally different than the initial training set.
To accommodate future tasks, we start learning with a larger one-hot vector.
Once a new task isdeﬁned and its success detector is trained, we allocate a next available one-hot task ID to this task.
To demonstrate this, we perform a ﬁne-tuning experiment, bootstrapping from the MT-Opt 12-taskpolicy (Sec. 5.2).
We use the MT-Opt policy to collect data for unseen tasks: lift-cloth and coverobject ( Fig. 7 bottom row).
Unlike lift-sausage , prior to starting collection of these new tasks, noepisodes in our ofﬂine dataset can be relabelled as successes for these tasks.
We follow the continuous data collection process (Sec. 4): we deﬁne and train the success detector for the new tasks, collect initialdata using our lift-any and a place-any policies, and ﬁne-tune a 14-task MT-Opt modelthat includes all prior as well as the newly deﬁned tasks.
While the new tasks are visuallyand semantically different, in practice the abovementioned policies give reasonable success ratenecessary to start the ﬁne-tuning.
We switchto running the new policies on the robots oncethey are at parity with the lift-any and placeany policies.
After 11 K pick-cloth attemptsand 3 K cover-object attempts (requiring < 1day of data collection on 7 robots), we obtainan extended 14-task MT policy that performscloth picking at 70% success and object covering at 44% success.
The policy trained onlyfor these two tasks, without support of our ofﬂine dataset, yields performance of 33% and5% respectively, conﬁrming the hypothesis that MT-Opt method is beneﬁcial even if the target tasks are sufﬁciently different, and the target data is scarce.
By collecting additional10 K pick-cloth episodes and 6 K cover-objectepisodes, we further increase the performanceof 14-task MT-Opt to 92% and 79% , for clothpicking and object covering respectively.
Weperform this ﬁne-tuning procedure with othernovel tasks such as previously unseen transparent bottle grasping, which reaches a performance of 60% after 4 days of collection. 5.7 Limitations of MT-Opt
While we show successful execution of MT-Opt on a number of tasks and its potential to scale, it isimportant to list limitations of this system. i) The shown tasks are short-horizon ( ≤ 20 steps). Longhorizon real-world robotic tasks are still hard for RL methods; ii)
The task outcome is determinedfrom the ﬁnal episode frame.
In theory, this could be extended to learned intermediate rewards; iii)
Our tasks leverage an automated reset mechanism. While this is not an assumption of the method,we have not demonstrated learning any reset-free behaviors; iv) Learning drastically different skillsis challenging, as our method works best when adapting more similar tasks. It remains an openquestion as to how far MT-Opt can be scaled to a very diverse set of skills; v) MT-Opt is 4DoF withgripper and terminate actions. Learning higher DoF tasks is challenging yet principally possible.
Conclusion
We presented a general multi-task learning framework, MT-Opt, that encompasses a number of elements: a multi-task data collection system, a scalable success detector framework and a multi-task deep RL method that is able to effectively utilize the multi-task data.
Our experiments show that by sharing data and parameters across various tasks we can signiﬁcantly increase the data efﬁciency of learning individual tasks.
Finally, we show how MT-Opt can quickly acquire new tasks by leverag- ing the shared multi-task representations and exploration strategies