On the Effectiveness of Fine-tuning Versus Meta-reinforcement Learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Intelligent agents should have the ability to leverage knowledge from previouslylearned tasks in order to learn new ones quickly and efﬁciently.
Meta-learningapproaches have emerged as a popular solution to achieve this.
However, metareinforcement learning (meta-RL) algorithms have thus far been restricted tosimple environments with narrow task distributions and have seen limited success.
Moreover, the paradigm of pretraining followed by ﬁne-tuning to adapt to new tasks has emerged as a simple yet effective solution in supervised learning.
This calls intoquestion the beneﬁts of meta learning approaches also in reinforcement learning,which typically come at the cost of high complexity.
We therefore investigate meta-RL approaches in a variety of vision-based benchmarks, including Procgen, RLBench, and Atari, where evaluations are made on completely novel tasks.
Our ﬁndings show that when meta-learning approaches are evaluated on different tasks(rather than different variations of the same task), multi-task pretraining with ﬁne- tuning on new tasks performs equally as well, or better, than meta-pretrainingwith meta test-time adaptation.
This is encouraging for future research, as multitask pretraining tends to be simpler and computationally cheaper than meta-RL.
From these ﬁndings, we advocate for evaluating future meta-RL methods on morechallenging tasks and including multi-task pretraining with ﬁne-tuning as a simple,yet strong baseline.
(cid:683) (cid:683) (cid:55)(cid:85)(cid:68)(cid:76)(cid:81)(cid:3) (cid:55)(cid:72)(cid:86)(cid:87) (cid:51)(cid:85)(cid:82)(cid:70)(cid:74)(cid:72)(cid:81) (cid:3) (cid:3)(cid:3)(cid:3) (cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:53)(cid:47)(cid:37)(cid:72)(cid:81)(cid:70)(cid:75) (cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:36)(cid:87)(cid:68)(cid:85)(cid:76)
[Figure]
Introduction
One of the major gaps between human and machine intelligence is the sample efﬁciency of learning.
Whereas humans can leverage past knowledge to learn a new task from a few examples, currentmachine learning systems often require a large amount of data and supervision to achieve even a single task.
To bridge this gap, meta-learning has become a popular approach — it uses many tasksto meta-train an optimal learning strategy, which enables few-shot generalization on a test task.
Meta-learning methods have had the most success in supervised learning settings [1–3], speciﬁcallyfew-shot classiﬁcation, where the goal is to learn a classiﬁer to recognize unseen classes during atest-time training phase with limited labeled data.
Recent work within the Computer Vision (CV)community has shown that variations of simple pretraining and ﬁne-tuning can perform equally aswell as more complex meta-learning approaches [4–7].
Why then, does simple multi-task pretrainingand ﬁnetuning not outperform meta-reinforcement learning (meta-RL)?
Our hypothesis is simple:the CV community evaluates their approaches on distinct test tasks (e.g. classifying dogs, cats,and birds), while the meta-RL community evaluates on variations of the same train-time tasks, forexample, treating varying transition dynamics (e.g. different friction parameters) or varying rewardfunctions (e.g. running forward v.s. running backward) as different tasks.
Variation adaptationis inherently easier than task adaptation, and does not paint a full picture of the shortcomings ofmeta-RL.
Moreover, meta-RL has only been studied in fully observable settings or with shapedrewards [8–10], neglecting more realistic real-world scenarios, where rewards are often sparse, andobservations are high-dimensional (e.g. images, point-clouds, etc).
Evidently, there is a gap in the literature, where a large scale study would be well placed to analyze thesetting of vision-based meta-RL across a truly diverse set of tasks.
We use 3 existing RL benchmarks:
Procgen [11], RLBench [12], and Atari Learning Environment (ALE) [13, 14] — each offers a diverseset of distinct tasks that we use for train and test.
For example in RLBench [12], an agent could betrained to pick up cups, take a USB out of computer, and reach target locations, while at test-time,adaptation would be evaluated on completely unseen tasks, such as lifting blocks and pushing buttons.
We investigate three prominent meta-RL algorithms of differing paradigms: Reptile [9] — a gradientbased method, PEARL [10] — a context-based method, and RL 2 [15] — an LSTM-based method.
Results from this study are enlightening: multi-task pretraining, followed by ﬁne-tuning on noveltasks, performs equally as well, or better, than all three meta-RL algorithms, while being muchsimpler and less computationally expensive to train.
In light of this, we advocate for future researchto shift towards more challenging benchmarks, and include multi-task pretraining with ﬁne-tuning asa simple, yet strong baseline.
To summarize, the key contributions of the study are as follows:
• We show that multi-task pretraining, followed by ﬁne-tuning on novel tasks, performs
equally as well, or better, than common meta-RL baselines.
To the best of our knowledge,
this has not been previously shown. • We perform the ﬁrst large-scale multi-task and meta-RL study of cross-game adaptation onAtari games.•
We perform the ﬁrst large-scale multi-task and meta-RL study of sparse reward manipulation tasks in RLBench.
• Within RLBench, we show that we can overcome sparse rewards on unseen test tasks vialarge-scale multi-task pretraining.
Preliminaries
Reinforcement Learning.
Reinforcement learning (RL) assumes access to an agent that interactswith an environment in which there are states s 2 S , actions a 2 A , and a reward function R ( s t , a t ) , 65 where t represents the current time step.
The agent must discover a policy ⇡ that maximizes theexpectation of the sum of discounted rewards, i.e., E ⇡ [ P t  t R ( s t , a t )] , where future rewards arediscounted by a factor  2 [0 , 1) .
Each policy ⇡ has a corresponding Q-value function Q ( s , a ) , whichdeﬁnes the agents expected return when following the policy after taking action a in state s .
Meta-Reinforcement Learning.
Meta-reinforcement learning (meta-RL) aims to quickly adapt toany task via RL.
Given a distribution of tasks p ( T ) , where each sampled task T i 2 p ( T ) , is a standalone MDP T i =
( S , A , P,  , R ) i , or partially observable MDP (POMDP) T i =
( S , A , P,  , R, O ) i , 72 in the case where an agent receives observations o 2 O rather than states.
Given a task T , the agent 73 is allowed to collect a small amount of data D T and use it to adapt the policy to obtain ⇡ T with an 74 RL algorithm.
Meta-RL training aims to ﬁnd an optimal initial policy or exploration strategy that maximizes the expected return of its adapted policies across all tasks within a limited number ofsteps: E T ⇠ p ( T ) , ( s t , a t ) ⇠ ⇡ T [ P t  t R ( s t , a t )] .
Assumption on Task Distribution.
We assume no ambiguity between any two tasks in the train ortest sets.
Namely, all tasks have distinct state-action spaces: 8 T i , T j 2 p ( T ) , i 6 = j, S i ⇥
A i \ S j ⇥
A j = ? .
An agent should be able to infer which task to perform based on a single-step observation(e.g.
whether it’s playing Breakout or Pong).
As a result, our multi-task pretraining baseline doesnot need a separate indicator to determine which task to complete.
Note that our assumption isdifferent from most prior work in meta-RL, which often assume the difference in “tasks” only occursin transition dynamics (e.g. different friction parameters) or the reward function (e.g. running forwardv.s. running backward).
2.1 Overview of Compared Algorithms
Reptile [9] is a ﬁrst-order gradient-based meta-learning algorithm.
It ﬁnds initial model parametersthat can be quickly adapted to any task via gradient descent.
Meta-training alternates betweenan inner loop — which performs multiple gradient steps on a single task, and an outer loop —which mixes the updated parameters from multiple tasks.
Reptile is mathematically similar to ﬁrstorder model-agnostic meta-learning [3] (MAML) while achieving comparable results as MAML inboth supervised [9] and imitation learning [16] settings.
We choose Reptile for its simplicity andapplicability to both on-policy and off-policy RL algorithms.
PEARL [10] is a context-based, off-policy meta-RL algorithm.
It meta-trains a policy that conditions on context embeddings from previous episode rollouts.
At test time, it adapts to a new taskby collecting new experiences to update the context, and adjusting its policy according to the newcontext.
RL 2 [15] is a recurrence-based, policy-gradient algorithm.
It represents the RL policy with arecurrent neural network (RNN) that embeds agent experiences from multiple “trials" in a single task,in the aim of encoding an RL update rule into the RNN weights.
At test time, it adapts by rolling outan agent in a new task, which updates the RNN hidden states and improves the policy to success aftera few trials.
Multi-task Training As a baseline, we run multi-task reinforcement learning (MTRL), whichsimply learns to complete all the training tasks simultaneously with a single RL agent.
At test time,the agent adapts to a new task by ﬁnetuning with the same RL algorithm as training.
Each of the above methods can be applied to a base RL algorithm .
For our Progen experiments(Section 2), we use PPO [17] as the base algorithm.
For our RLBench experiments (Section 4), we use C2F-ARM [18] as the base.
Finally, for our Atari experiments (Section 6), we use RainbowDQN [19]as the base algorithm.
Justiﬁcation for each of these choices is given in their respective sections.
See Algorithms 1 and 2 in Appendix A for a uniﬁed overview of how the meta-RL algorithms areimplemented.
All experiments in the study are trained on a maximum of 8 RTX A5000 GPUs, each with 24GB ofmemory.
Unless otherwise indicated, we use the same hyperparameters and architectures as used inprior work; full details are given in the appendix.
Procgen Experiments
Procgen Benchmark [11] is a suite of procedurally-generated game-like environments designed forstudying generalization in deep RL, where an agent is trained on a set of levels and evaluate on unseenlevels.
We classify Procgen as the easiest benchmark in this study, because both the embodiment andtask objects are consistent across tasks, while only the colors and layouts vary.
Prior work has shown an improvement to generalization by training on a large amount oflevels [11] or adding data-augmentation to visual inputs [20, 21].
Beyond zero-shot generalization, few prior work use Procgen to evaluate meta-RL, with one exception being Alver et al.
[22], which showed that RL 2 failed to generalize on simpliﬁed Procgen games.
Figure is trained tested on 20 unseen levels.
We choose one representative Procgen game, namely Coinrun, where an agent is tasked to avoid obstaclesand reach a golden coin.
The game contains multiple“levels”, each of which has a unique color theme orlayout.
We use up to 10,000 levels for training, andheld-out 20 levels for testing. 100 million environmentsteps are used for all pretraining runs, and we the checkpoint adaptation environment run
3.1 Training Setup
We use PPO [17] as the base RL algorithm and use thesame image encoder as IMPALA [23], which has beenshown to achieve competitive performance on singlegame settings in Procgen [11, 24].
Note that PEARL isexcluded from this section because it is off-policy.
Reptile-PPO incorporates the inner-outer loop in Reptile [9] with PPO.
For each iteration in the inner loop,the environment is ﬁxed to one randomly sampled tasklevel, where rollouts are collected and used to perform3 iterations of batched gradient updates.
Following that,the outer loop takes the updated policy parameters, and perform a soft parameter update.
This processis repeated iteratively.
RL 2 -PPO combines an LSTM with PPO as proposed in [15].
We modify the Coinrun environment tore-sample the same level for multiple episodes, such that one original environment episode is treatedas one “trial”, and each RL 2 episode contains multiple “trials”.
Until an episode is done, the agentspast rollouts are concatenated and fed into an LSTM layer.
The image observations are ﬁrst encodedinto 1-dimensional latent vectors, then concatenated with action and reward, and embedded againbefore they are fed into LSTM hidden layers.
MT-PPO jointly trains PPO on multiple levels of Coinrun.
The environment randomly samplesfrom a ﬁxed number of training levels, and the policy is updated with rollouts from multiple levels.
3.2 Test-time Adaptation Setup
For Reptile-PPO and MT-PPO , atrained agent is ﬁnetuned with vanilla PPO for 2 million environment stepson each test level.
For RL 2 -PPO , the LSTM hidden states are reset, and theagent is rolled-out on each new levelfor 2 million environment steps without gradient update.
We also compareadaptation with training from scratch,where one agent is trained for eachtest level with PPO for 2 million steps.
We use a total of 20 held-out levelsfor testing, and results are reported byaveraging across all 20 levels.
3.3 Results
Results for testing on unseen levels are shown in Figure 3.
We ﬁrst remark that, ﬁnetuning achievesthe best performance at adaptation to new levels both in terms of sample efﬁciency and ﬁnalperformance, and is shown consistently across multiple numbers of training levels.
Reptile-PPO also improves slightly after ﬁnetuning, but the performance is worse than MT-PPO, which indicates thelearned parameter initialization is slightly worse.
RL 2 does not improve performance on unseen levels, meaning that the updated hidden state of the
RNN fails to adapt sufﬁciently while testing on new levels.
This is consistent with [22], where RL 2fails to adapt even after simplifying the level such that the coin is visible at the beginning of eachtrial.
Figure 3 also shows that increasing the number of training levels improves zero-shot performanceon MT-PPO (and less so for Reptile-PPO), which is consistently with prior work on generalizationin RL.
We demonstrate that expanding the training set also beneﬁts ﬁnetuning and provides a clearadvantage over training from scratch, which is encouraging news for simple scaling of training tasksas an alternative to designing computationally complex meta-training algorithms.
RLBench Experiments
RLBench is a vision-based manipulation benchmark and learning environment, with a focus on sparserewards, multi-task learning, and meta-learning.
The environment has a rich set of more than 100real-world inspired tasks involving diverse objects.
The environment also provides easy access toexpert demonstrations for all tasks, which is vital for overcoming the exploration problem imposedby the benchmark’s sparse rewards.
We classify RLBench as the medium difﬁculty benchmark in thisstudy; while the embodiment is consistent across tasks (i.e., the Franka Panda robot arm), the taskobjects and goals vary drastically.
To ensure the experiment results do not get affected by arbitrary task selection, we design acomprehensive set of train-test task splits that resembles cross-validation in supervised learning setting.
Speciﬁcally, we use a ﬁxed set of 11 RLBench tasks, and create 5 splits.
Eachsplit uses a (randomly selected) held-out task and trains an agent on the remaining 10 tasks.(cid:20)(cid:19)(cid:3)(cid:55)(cid:68)(cid:86)(cid:78)(cid:3) (cid:171) (cid:55)(cid:72)(cid:86)(cid:87) (cid:55)(cid:85)(cid:68)(cid:76)(cid:81) [Figure]
4.1 Training Setup
[Figure] We use C2F-ARM [18] as the base off-policy
RL algorithm. This was chosen because morewidely-used RL algorithm, such as DDPG [25], TD3 [26], SAC [27], and DrQ [21] are knownto fail [28] in RLBench due to the challengingsetup.
C2F-ARM [18] is a vision-based robotmanipulation algorithm that can learn sparsereward reinforcement learning task by using asmall number of initial demonstrations.
C2FARM is described in more detail in Appendix C. Note that RL 2 is excluded from this section because it is on-policy.
Reptile-C2F-ARM modiﬁes the off-policy batch update in C2F-ARM with an inner- and outer-loopproposed in Reptile [9].
At the beginning of training, each task is given a separate replay buffer,which is initialized with transitions collected from 5 demonstration trajectories and continouslyappended with the agent’s online experiences.
During training, for multiple steps in the inner loop,the agent draws a batch from the replay buffer of a randomly sampled task, and performs updates tothe Q-attention.
In the outer loop, the network gets a soft update to mix the parameters from beforeand after the inner loop updates.
PEARL-C2F-ARM conditions a context embedding to the Q-attention network.
To obtain thecontext for a task, a batch of transitions is drawn from a window of recent agent experiences, and aseparate convolution encoder is used for ﬁrst encode the image observations individually.
Then, eachimage embedding is concatenated with the action and reward, and together encoded into a singlevector.
Finally the context embeddings are sampled as proposed
4.2 Test-time Adaptation Setup
Both MT-C2F-ARM and Reptile-C2F-ARM use the same C2F-ARM update and adapt the agentparameters to the new task via gradient descent.
Adaptation for PEARL-C2F-ARM is done bygathering rollout samples in the new environment and re-computing the context embeddings, hencerunning only inference on the agent’s policy model.
4.3 Results
The ﬁrst set of evaluations are the most challenging for adaptation: an unseen test-time task given0 demonstrations .
This setup is designed to test the agent’s ability to leverage knowledge andskills gained in the 10 training tasks in order to perform intelligent exploration on the test task,without any guidance from demonstrations.
Results for this challenge are presented in the top row of Figure 5.
Across all 5 test tasks, multi-task ﬁne-tuning performs equally as well as gradient-basedmeta-learning, while performing signiﬁcantly better than both context-based algorithms and trainingfrom scratch.
We next investigate how much of an effect the reward sparsity has on the test-time performance.
Toaid with test-time exploration, we now provide test-time demonstrations to each of the methods.
Theinvestigation in the second and third row of Figure 5 investigates how the methods behave when given1 and 2 test-time demonstrations .
The fact that increasing the number of demonstrations improvestraining from scratch performance is unsurprising, however, one intriguing observation is that thiseffect is less apparent for MT-C2F-ARM and Reptile-C2F-ARM methods.
This is encouraging forthe robotics community (and communities where sparse rewards are prevalent): although we mayneed demonstrations for pretraining, we can signiﬁcantly reduce (or even omit) demonstrationsin the ﬁne-tuning phase, with little loss to performance.
With these result in mind, Appendix Cinvestigates further the various properties of ﬁne-tuning C2F-ARM.
Apparent from Figure 5 is the fact that PEARL does not seem equipped to handle such a disjointtrain-test split.
This is not surprising given its high reliance on context for adaptation: the agent modelparameters stay ﬁxed, and the only way to understand a new task is via aggregating new experiencesinto the context.
However, it is clear that the context encoder fails at unseen tasks; we hypothesizethe reason lies in our tasks setup: during training, the tasks are so visually disjoint that the agentnever needs to read a context embedding to infer which task it should do.
Recall, that meta-RLalgorithms usually treat variations as “tasks”, meaning that the different “tasks” look visually similar,or even identical; in order to disentangle the required task, the network is heavily motivated to readthe context. However, when tasks are disjoint, and there is a clear visual difference between tasks, thecontext becomes redundant, and the agent is no longer motivated to read the context during training.
Further evidence towards this hypothesis can be seen by looking at the zero-shot performance of the PEARL agent (i.e., environment steps = 0 ), where it starts with the same performance as multi-taskand Reptile agents.
However, as the ﬁne-tuning process progresses, PEARL performance stagnates.
This suggests the meaningful performance that PEARL does achieve should be attributed to thepretraining and not test-time adaptation.
Atari Experiments
The Arcade Learning Environment (ALE, or the Atari benchmark) is one of the most commonly usedenvironments within the RL community; however, rather than using the benchmark to train individualagents on each game, we use it to study meta-RL.
We classify Atari as the hardest benchmark in thisstudy due to the fact that the embodiment, objects, and goals, all vary across tasks; i.e., there is verylittle overlap between two different games.
Prior work has studied transfer between Atari games in relative small scale, such as training an RLagent on one game and transfer to a visually similar game via ﬁne-tuning [29, 30].
To the best of ourknowledge, no prior meta-RL algorithm has been meta-trained on a subset of Atari games and testedon other games.
Recently, Oh et al.
[31] demonstrated meta-training via discovering RL update rulescan transfer from mini-grid environment to Atari, but performance still falls short from state-of-theart results on training Atari games from scratch. 5.0.1 Training Setup
We use the data-efﬁcient version of RainbowDQN [19] as the base RL algorithm, i
[Table]
e. wethe the same training hyper-parameters and image encoder network architecture as proposed in [32].
Note that RL 2 is excluded from this section because it is on-policy.
[Table] (cid:24)(cid:3)(cid:55)(cid:68)(cid:86)(cid:78)(cid:3)(cid:55)(cid:85)(cid:68)(cid:76)(cid:81) [Table]
Reptile-Rainbow incorporates Reptile [9] withthe off-policy batch update in Rainbow.
Similar to Reptile-C2F-ARM, transitions from each Atari game are stored in a separate replay buffer.
For multiple steps in the inner loop, the agentdraws a batch from a randomly sampled bufferand performs updates to the distributional valuenetwork.
In the outer loop, the value networkgets a soft update to mix parameters before andafter the inner loop updates.
PEARL-Rainbow conditions a context embedding to the value network.
During each batchupdate, samples from each task are drawn froma recent window of agent experiences and getencoded as the context.
The context encoder isadditionally trained with a KL loss [10].
MT-Rainbow jointly trains RainbowDQN onall training games/tasks.
In contrast to ReptileRainbow, the replay batch in both MT-Rainbow and PEARL-Rainbow contains samples from multipletask replay buffers.
During each replay update, we randomly select a ﬁxed number of tasks (less or equal to the total number of avaliable training tasks), then draw an equal number of samples for eachtask and construct the replay batch.
Note that, Atari games generally have different action dimensions, hence for all pretrained agents,the action space is padded to be the maximum possible size (discrete, 18-dimensional).
The extradimensions are all mapped to “No-op" for the games with smaller original action space.
For trainingfrom scratch, the agents are trained on the original action dimensions on their corresponding testgame.
5.1 Test-time Adaptation Setup
We use 5 test-time tasks, namely Pong, Assault, Battle Zone, Ms Pacman, and Beam Rider.
For
Reptile-Rainbow and MT-Rainbow, a trained agent is ﬁnetuned with Rainbow for 100,000 environment steps on each unseen game, following the data-efﬁcient benchmark proposed in [32].
For
PEARL-Rainbow, the agent is rolled-out on a new game for 100,000 environment steps, and onlyupdates the context embedding by encoding newly collected experiences.
For both Reptile-Rainbow and MT-Rainbow, all network parameters are ﬁnetuned.
We provideadditional results in Appendix D for loading only the convolutional encoder and re-initializing all
MLP layers during ﬁnetuning.
This setup was previously investigated in [30] at a smaller scale,i.e. training a Breakout agent and ﬁne-tuning on Pong, but the network architecture and base RLalgorithms differ from ours.
5.2 Results
We report the test-time adaptation performance for each pretrained agent in Figure 7.
Theresults are divided into 3 sets, based on the number of the training tasks provided toeach compared algorithm.
For each set of experiments, the same training from “scratch”baseline is used (colored in gray).
We make the following observations and remarks:
For 5-task- and 10-task- trained agents ,we observe high variance across differenttest-time games despite being ﬁnetunedfrom the same pretrained agent.
For example, ﬁnetuning the Reptile-Rainbow agenttrained on 10 tasks provides a clear advantage over training from scratch on Assault,but falls short on Ms Pacman and Beamrider.
Consistent with results on RLBench,
PEARL-Rainbow also fails to adapt to theunseen task despite continuously updatingits context embedding for a sufﬁcientlylong period.
PEARL-Rainbow’s zero-shotperformance on some of the games isclearly better than acting randomly (i.e. theﬁrst datapoint for training from scratch)— however, this beneﬁt comes more from the diversity oftraining tasks than from the meta-training algorithm itself, which is evident from observing that: (1) PEARL-Rainbow has similar zero-shot performance to that of MT-Rainbow and Reptile-Rainbow,and (2) as the number of training task goes from 5 to 10, PEARL-Rainbow’s performance slightlyimproves.
Overall, training from scratch has competitive performance across all test-time tasks. This is incontrast to previous results on Procgen and RLBench, and calls for a closer inspection at thetransferability between Atari games.
Related Work
Meta-Reinforcement Learning Meta-RL aims to ﬁnd the best learning strategy that enablesfast adaptation to a new task via reinforcement learning.
This often relies on meta-training witha distribution of tasks and exploiting their shared structures.
The area can be broadly separatedinto context-based methods and gradient-based methods.
Context-based methods are trained to userecent rollout experiences from a new task to form a context that can be used to distinguish whattask the policy is solving.
Previously, this context has been formed implicitly via an LSTM [15,33], or explicitly, by passing trajectories through a separate encoder, whose output is given to acontext-conditioned policy [10, 34].
Gradient-based methods perform test-time optimisation ofhyperparameters [35], loss functions [36, 37], or network parameters [3, 38, 39].
The meta-RL approaches above have only been studied in fully observable state settings withshaped rewards; neglecting more realistic real-world scenarios, where rewards are often sparse, andobservations are high-dimensional (e.g. images, point-clouds, etc).
There is limited work where theseissues have been studied independently, e.g. [40], where hindsight relabeling is used to enable learningto learn entirely using sparse reward; however this relies on fully observable state.
Vision-basedmeta-learning outside of CV has been applied to robot manipulation [8, 41, 42], however this is posedas a imitation learning problem.
Multi-task Reinforcement Learning The pretraining procedure in our experiments is simplymulti-task reinforcement learning (MTRL), where the training objective is ﬁnding a single best policyacross multiple tasks and does not prepare for adaptation.
The main challenge in multi-task learningin general lies in multi-objective optimization, and has been investigated in RL [43, 44] and appliedto robotics [45].
Although various multi-task algorithms has been proposed, Kurin et al.
[46] hasdemonstrated that joint training with proper regularization achieves competitive performance withthe more complicated methods.
This observation aligns with our multi-task training results
.Meta- v.s. Multi-task pretraining in RL
Recent work has looked into the comparison betweenjoint training and gradient-based meta-RL algorithms (e.g. MAML), and shows both theoreticallyand empirically that ﬁne-tuning after joint training achieves comparable performance [47].
Becausethe training set in popular meta-RL benchmarks are mainly variations of the same task (e.g. differentcheetah running direction), the joint training setup has also been called domain random search(DRS) [48]. Our work further expands on this setup by training on more distinct tasks instead ofvariations, and the test-time task is strictly excluded from training.
Conclusion
We performed a large-scale study to analyze the setting of vision-based meta-RL, across a trulydiverse set of tasks.
This was in an attempt to see if we could replicate the ﬁndings within thecomputer vision community — that simple pretraining and ﬁne-tuning can perform equally as well asmore complex meta-learning approaches [4–7]. A large body of literature has already shown thatmeta-RL is effective when evaluated on variations of the same train-time tasks, however, we haveshown that when meta-RL is tested on truly diverse reinforcement learning tasks, the ﬁndings withinthe CV community hold true.
Our work is an initial step towards understanding the subtleties between meta-RL and multi-taskpretraining, with plenty of room for further investigation.
Despite the breadth of our experimentsacross 3 benchmarks, a limitation of our study is the limited number of pre-training tasks for both
RLBench and Atari.
This may not be sufﬁcient to fully learn the representations required to enablethe best adaptation performance on new tasks.
However, note that the purpose of this study was notto show that multi-task pretraining followed by ﬁne-tuning is better than training from scratch, butrather that multi-task pretraining can perform equally as well, or better, than meta-RL.
Our paperis a study of existing methods, and therefore foresee a low risk of negative societal impacts. Ourstudy may increase interest in multi-task pretraining — an area that often requires a large amount ofcompute for training, the community should therefore be cautious of the environmental impact. Checklist For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope? [Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]