On the Effectiveness of Fine-tuning Versus Meta-reinforcement Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
Intelligent agents should have the ability to leverage knowledge from previouslylearned tasks in order to learn new ones quickly and efficiently.
Meta-learningapproaches have emerged as a popular solution to achieve this.
However, metareinforcement learning (meta-RL) algorithms have thus far been restricted to simpleenvironments with narrow task distributions.
Moreover, the paradigm of pretrainingfollowed by fine-tuning to adapt to new tasks has emerged as a simple yet effectivesolution in supervised and self-supervised learning.
This calls into question thebenefits of meta-learning approaches in reinforcement learning, which typicallycome at the cost of high complexity.
We therefore investigate meta-RL approachesin a variety of vision-based benchmarks, including Procgen, RLBench, and Atari,where evaluations are made on completely novel tasks.
Our findings show that whenmeta-learning approaches are evaluated on different tasks (rather than differentvariations of the same task), multi-task pretraining with fine-tuning on new tasksperforms equally as well, or better, than meta-pretraining with meta test-timeadaptation.
This is encouraging for future research, as multi-task pretraining tendsto be simpler and computationally cheaper than meta-RL.
From these findings, weadvocate for evaluating future meta-RL methods on more challenging tasks andincluding multi-task pretraining with fine-tuning as a simple, yet strong baseline.
[Figure]
Introduction
One of the major gaps between human and machine intelligence is the sample efficiency of learning.
Whereas humans can leverage past knowledge to learn a new task from a few examples, currentmachine learning systems often require a large amount of data and supervision to achieve even asingle task.
To bridge this gap, meta-learning has become a popular approach — it uses many tasksto meta-train an optimal learning strategy, which enables few-shot generalization on a test task.
Meta-learning methods have had the most success in supervised learning settings [1, 2, 3], specificallyfew-shot image classification, where the goal is to learn a classifier to recognize unseen classesduring a test-time training phase with limited labeled data.
Recent work has found that variationsof simple pretraining and fine-tuning can perform equally as well as more complex meta-learningapproaches [4, 5, 6, 7].
However, within the realm of reinforcement learning, simple pretraining andfine-tuning is not known to out perform meta-reinforcement learning (meta-RL).
Our hypothesisfor this discrepancy is simple: the computer vision (CV) community evaluates their approaches ondistinct test tasks (e.g. classifying dogs, cats, and birds), while the meta-RL community evaluateson variations of the same train-time tasks; for example, varying transition dynamics (e.g. differentfriction parameters) or varying reward functions (e.g. running forward v.s. running backward)are better categorized as variations rather than different tasks, as discussed in recent work [8, 9].
Variation adaptation is inherently easier than task adaptation, and does not paint a full picture of theshortcomings of meta-RL.
Moreover, most meta-RL methods (with a few exceptions, discussed in 6)have been studied in fully observable settings or with shaped rewards [10, 11, 12], neglecting morerealistic real-world scenarios, where rewards are often sparse, and observations are high-dimensional(e.g. images, point-clouds, etc).
Evidently, there is a gap in the literature, where a large-scale study would be well placed to analyze thesetting of vision-based meta-RL across a truly diverse set of tasks.
We use 3 existing RL benchmarks:
Procgen [13], RLBench [8], and Atari Learning Environment (ALE) [14, 15] — each offers a diverseset of distinct tasks that we use for train and test.
For example in RLBench [8], an agent could betrained to pick up cups, take a USB out of a computer, and reach target locations, while at test time,adaptation would be evaluated on completely unseen tasks, such as lifting blocks and pushing buttons.
We investigate three prominent meta-RL algorithms of differing paradigms: Reptile [11] — a gradientbased method, PEARL [12] — a context-based method, and RL 2 [16] — an LSTM-based method.
Results from this study are enlightening: multi-task pretraining, followed by fine-tuning on noveltasks, performs equally as well, or better, than all three meta-RL algorithms, while being muchsimpler and less computationally expensive to train.
In light of this, we advocate for future researchto shift towards more challenging benchmarks, and include multi-task pretraining with fine-tuning asa simple, yet strong baseline.
To summarize, the key contributions of our study are as follows:
• We show that multi-task pretraining followed by fine-tuning on novel tasks performs equally as
well, or better, than common meta-RL baselines on vision-based environments.
• We present the first large-scale multi-task and meta-RL study on three existing benchmarks:cross-level adaptation on Procgen, cross-task adaptation on RLBench, and cross-game adaptation
on Atari.
• Within RLBench, we show that large-scale multi-task pretraining can overcome sparse rewardson unseen test tasks and perform significantly better than training from scratch .
Preliminaries
Reinforcement Learning.
Reinforcement learning (RL) assumes access to an agent that interactswith an environment in which there are states s ∈ S , actions a ∈ A , and a reward function R ( s t , a t ) ,where t represents the current time step.
The agent must discover a policy π that maximizes theexpectation of the sum of discounted rewards, i.e., E π [ (cid:80) t γ t R ( s t , a t )] , where future rewards arediscounted by a factor γ ∈ [0 , 1) .
Each policy π has a corresponding Q-value function Q ( s , a ) , whichdefines the agents expected return when following the policy after taking action a in state s .
Meta-Reinforcement Learning.
Meta-reinforcement learning (meta-RL) aims to quickly adaptto any task via RL.
Given a distribution of tasks p ( T ) , where each sampled task T i ∈ p ( T ) ,is a stand-alone MDP T
i = ( S , A , P, γ, R ) i , or partially observable MDP (POMDP) T i =
( S , A , P, γ, R, O ) i , in the case where an agent receives observations o ∈ O rather than states.
Given a task T , the agent is allowed to collect a small amount of data D T and use it toadapt the policy to obtain π T with an RL algorithm.
Meta-RL training aims to find an optimal initial policy or exploration strategy that maximizes the expected return of its adapted policies across all tasks within a limited number of steps: E T ∼ p ( T ) , ( s t , a t ) ∼ π T [ (cid:80) t γ t R ( s t , a t )] . [Figure]
Assumption on Task Distribution.
We assume no ambiguity between any two tasks inthe train or test sets.
Namely, all tasks havedistinct, non-overlapping state-action spaces:∀T i [Equation] ) , i
[Equation] j, S i
[Equation]
An agent should be able to infer which taskto perform based on a single-step observation(e.g.
whether it’s playing Breakout or Pong).
As a result, our multi-task pretraining baselinedoes not need a separate indicator to determinewhich task to complete.
Note how this assumption is different from most prior work in metaRL, which often assume the difference in “tasks”only occurs in transition dynamics (e.g. different friction parameters) or the reward function(e.g. running forward v.s. running backward) —hence we categorize those prior assumptions as“variation adaptation" to better distinguish fromour work, as illustrated in Figure 2.
2.1 Overview of Compared Algorithms
Reptile [11] is a first-order gradient-based meta-learning algorithm.
It finds initial model parametersthat can be quickly adapted to any task via gradient descent.
Meta-training alternates between aninner loop — which performs multiple gradient steps on a single task, and an outer loop — whichmixes the updated parameters from multiple tasks.
Reptile is mathematically similar to first-ordermodel-agnostic meta-learning [3] (MAML) while achieving comparable results as MAML in bothsupervised [11] and imitation learning [17] settings.
We choose Reptile for its simplicity andapplicability to both on-policy and off-policy RL algorithms.
PEARL [12] is a context-based, off-policy meta-RL algorithm.
It meta-trains a policy that conditions on context variables, which are encoded from small amounts of an agent’s experiences and areused to perform online probabilistic filtering to infer how to solve a new task.
At test time, it adaptsto a new task by collecting new experiences to update the context and adjusting its policy accordingto the new context.
RL 2 [16] is a recurrence-based, policy-gradient algorithm.
It represents the RL policy with arecurrent neural network (RNN) that embeds agent experiences from multiple “trials” in a single task,with the aim of encoding an RL update rule into the RNN weights.
At test time, it adapts by rollingout an agent in a new task, which updates the RNN hidden states and improves the policy to successafter a few trials.
Multi-task Training As a baseline, we run multi-task reinforcement learning (MTRL), whichsimply learns to complete all the training tasks simultaneously with a single RL agent.
At test time,the agent adapts to a new task by finetuning with the same RL algorithm as training.
Base RL Algorithms Each of the above methods is applied to a base RL algorithm that are differentacross benchmarks.
For our Progen experiments (Section 3), we use PPO [18].
For our RLBenchexperiments (Section 5), we use C2F-ARM [19].
For our Atari experiments (Section 7), we use RainbowDQN [20] as the base algorithm.
Justification for each of these choices is given in theirrespective sections.
See Algorithms 1 and 2 in Appendix B for a unified overview of how the meta-RLalgorithms are implemented.
All experiments in the study are trained on a maximum of 8 RTX A5000 GPUs, each with 24GB ofmemory.
Unless otherwise indicated, we use the same hyperparameters and architectures as used inprior work; full details are given in the appendix.
Procgen Experiments
[Figure]
Train
[Figure]
Train … … … … … … …… … …
Test … … [Figure] … 1000 levels … Procgen Benchmark [13] is a suite of procedurallygenerated game-like environments designed for studyinggeneralization in deep RL, where an agent is trained on aset of levels and evaluated on unseen levels.
We classify Procgen as the easiest benchmark in this study, becauseboth the embodiment and task objects are consistent acrosstasks, while only the colors and layouts vary.
Prior work has explored improving generalization bytraining on a large amount of levels [13], adding dataaugmentation to visual inputs [21, 22], knowledge transferduring training [23], and self-supervised world models[24].
Beyond zero-shot generalization, few prior worksuse Procgen to evaluate meta-RL, with one exception being Alver et al.
[25], which showed that RL 2 failed togeneralize on simplified Procgen games.
We choose one representative Procgen game, namely Coinrun, where an agent is tasked to avoid obstacles and reacha golden coin.
The game contains multiple “levels”, eachof which has a unique color theme or layout.
We use up to10,000 levels for training and held-out 20 levels for testing.100 million environment steps are used for all pretrainingruns, and we take the model checkpoint after 1500 PPOiterations for testing adaptation — the environment is runin parallel on 256 threads and each PPO iteration uses 256steps.
3.1 Training Setup
We use PPO [18] as the base RL algorithm and use the same image encoder as IMPALA [26], whichhas been shown to achieve competitive performance on single-game settings in Procgen [13, 27].
Note that PEARL is excluded from this section because it is off-policy.
Reptile-PPO incorporates the inner-outer loop in Reptile [11] with PPO.
For each iteration in theinner loop, the environment is fixed to one randomly sampled task level, where rollouts are collectedand used to perform 3 iterations of batched gradient updates.
Following that, the outer loop takes theupdated policy parameters, and performs a soft parameter update.
This process is repeated iteratively.
RL 2 -PPO combines an LSTM with PPO as proposed in [16].
We modify the Coinrun environment tore-sample the same level for multiple episodes, such that one original environment episode is treatedas one “trial”, and each RL 2 episode contains multiple “trials”.
Until an episode is done, the agent’spast rollouts are concatenated and fed into an LSTM layer.
The image observations are first encodedby the IMPALA (following [13]) and flattened into 256-dimensional latent vectors, then concatenatedwith action, reward, and done signals. The concatenated vectors are embedded again before fed into LSTM hidden layers.
MT-PPO jointly trains PPO on multiple levels of Coinrun.
The environment randomly samples froma fixed number of training levels, and the policy is updated with rollouts from multiple levels.
3.2 Test-time Adaptation Setup
For Reptile-PPO and MT-PPO , a trained agent is finetuned with vanilla PPO for 2 million environment steps on each test level.
For RL 2 -PPO , the LSTM hidden states are reset, and the agentis rolled-out on each new level for 2 million environment steps without gradient updates.
We alsocompare adaptation with training from scratch, where one agent is trained for each test level with PPO for 2 million steps.
We use a total of 20 held-out levels for testing, and results are reported byaveraging across all 20 levels.
3.3 Results
[Figure] Results for testing on unseen levels are shownin Figure 4.
We first remark that finetuningachieves the best performance at adaptationto new levels both in terms of sample efficiencyand final performance, and is shown consistentlyacross varying number of training levels.
Notably, Reptile-PPO is also able to improve performance on test tasks, but the finetuning performance is worse than MT-PPO, which suggeststhe Reptile-learned parameters provides a lessadaptable initialization than the simpler multitask trained parameters.
Figure 4 also showsthat increasing the number of training levels improves zero-shot performance on MT-PPO (andless so for Reptile-PPO), which is consistent with the aforementioned prior work that studys generalization on Procgen.
Expanding the training set also benefits finetuning and provides a clear advantageover training from scratch, which is encouraging news for simple scaling of training tasks as analternative to designing computationally complex meta-training algorithms.
RL 2 does not improve performance on unseen levels, meaning that the updated hidden state of the
RNN fails to adapt sufficiently while testing on new levels.
This is consistent with [25], where RL 2fails to adapt even after simplifying the level such that the coin is visible at the beginning of eachtrial.  RLBench Experiments [Figure]
Train
Train [Figure] Test
[Figure] [Figure] Train Figure 5:
In RLBench experiments, each agent is trained on 10 tasks and evaluated on 5 unseen tasks.
RLBench is a vision-based manipulation benchmark and learning environment, with a focus onsparse rewards, multi-task learning, and metalearning.
The environment has a rich set ofmore than 100 real-world inspired tasks involving diverse objects.
The environment alsoprovides easy access to expert demonstrationsfor all tasks, which is vital for overcomingthe exploration problem imposed by the benchmark’s sparse rewards.
We classify RLBench asthe medium difficulty benchmark in this study;while the embodiment is consistent across tasks(i.e. , the Franka Panda robot arm), the task objects and goals vary drastically.
To ensure the experiment results do not get affected by arbitrary task selection, we design acomprehensive set of train-test task splits that resemble cross-validation in the supervised learningsetting.
Specifically, we use a fixed set of 11 RLBench tasks and create 5 splits.
Each split uses a(randomly selected) held-out task and trains an agent on the remaining 10 tasks.
4.1 Training Setup
We use C2F-ARM [19] as the base off-policy RL algorithm.
This was chosen because more widelyused RL algorithm, such as DDPG [28], TD3 [29], SAC [30], and DrQ [22] are known to fail [31]in RLBench due to the challenging setup.
C2F-ARM [19] is a vision-based robot manipulationalgorithm that can learn sparse-reward reinforcement learning tasks by using a small number of initialdemonstrations.
C2F-ARM is described in more detail in Appendix D. Note that RL 2 is excludedfrom this section because it is on-policy.
Reptile-C2F-ARM modifies the off-policy batch update in C2F-ARM with an inner- and outer-loopproposed in Reptile [11].
At the beginning of training, each task is given a separate replay buffer,which is initialized with transitions collected from 5 demonstration trajectories and continuouslyappended with the agent’s online experiences.
During training, for multiple steps in the inner loop, the agent draws a batch from the replay buffer of a randomly sampled task and performs updates tothe Q-attention.
In the outer loop, the network gets a soft update to mix the parameters from beforeand after the inner loop updates.
PEARL-C2F-ARM conditions a context embedding to the Q-attention network.
To obtain thecontext for a task, a batch of transitions is drawn from a window of recent agent experiences, and aseparate convolution encoder is used to first encode the image observations individually.
Then, eachimage embedding is concatenated with the action and reward, and together encoded into a singlevector.
Finally, the context embeddings are sampled as proposed in [12].
The context encoder isadditionally trained with a KL loss.
MT-C2F-ARM jointly trains C2F-ARM on all training tasks.
During each replay batch update, both
MT-C2F-ARM draw samples from multiple task replay buffers.
During each replay update, a fixednumber of tasks (less or equal to the total number of available training tasks) are randomly selected,then an equal number of samples are drawn for each task to construct the replay batch.
4.2 Test-time Adaptation Setup
Both MT-C2F-ARM and Reptile-C2F-ARM use the same C2F-ARM update and adapt the agentparameters to the new task via gradient descent.
Adaptation for PEARL-C2F-ARM is done bygathering rollout samples in the new environment and re-computing the context embeddings, hencerunning only inference on the agent’s policy model.
4.3 Results
The first set of evaluations are the most challenging for adaptation: an unseen test-time task given 0demonstrations .
The agent is expected to leverage knowledge and skills gained in the 10 trainingtasks and perform intelligent exploration on the test task, without any guidance from demonstrations.
Results for this setup are presented in the top row of Figure 6.
Across all 5 test tasks, multi-taskfine-tuning performs equally as well as Reptile while performing significantly better than both PEARLand training from scratch.
We next investigate the effect of reward sparsity on test-time performance.
We now provide test-timedemonstrations of each of the methods, as an aid for exploration under sparse reward.
Results inthe second and third row of Figure 6 show how the methods behave when given 1 and 2 test-timedemonstrations .
The fact that increasing the number of demonstrations improves training from scratchperformance is unsurprising, however, one intriguing observation is that this effect is less apparentfor MT-C2F-ARM and Reptile-C2F-ARM methods.
This is encouraging evidence that fine-tuningsignificantly reduces (or even omit) the need for demonstrations in sparse rewarded tasks, withlittle loss to performance.
We further investigate the various properties of fine-tuning C2F-ARM in Appendix D.
Apparent from Figure 6 is that PEARL does not seem equipped to handle such a disjoint train-testsplit.
Recall that PEARL adapts without model parameter updates, and the only way to understand anew task is via aggregating new experiences into the context.
However, the context encoder clearlyfails at providing a useful context for the unseen tasks. we hypothesize this is due to our tasks setup:the training tasks are so visually disjoint that the agent never needs to learn high-quality contextembeddings to infer which task it should do.
This is different from the original experiment setupin PEARL, where variations are treated as “tasks”, meaning that the observations from different“tasks” are similar or even identical; in order to disentangle the correct task to perform, the networkis heavily motivated to read the context.
Further evidence towards this hypothesis can be seen bylooking at the zero-shot performance of the PEARL agent (i.e., environment steps = 0 ), where itstarts with the same performance as multi-task and Reptile agents but doesn’t improve.
This suggeststhe meaningful performance that PEARL does achieve should be attributed to the pretraining and nottest-time adaptation.
Atari Experiments
The Arcade Learning Environment (ALE, or the Atari benchmark) is one of the most commonly usedenvironments within the RL community; however, rather than using the benchmark to train individualagents on each game, we use it to study meta-RL.
We classify Atari as the hardest benchmark in this [Figure] study due to the fact that the embodiment, objects, and goals, all vary across tasks; i.e., there is verylittle overlap between two different games.
Prior work has studied transfer between Atari games on relatively small scales, such as training an RL agent on one game and transfer to a visually similar game via fine-tuning [32, 33].
To the bestof our knowledge, no prior meta-RL algorithm has been meta-trained on a subset of Atari gamesand tested on other games.
Recently, Oh et al.
[34] demonstrated meta-training via discovering RLupdate rules can transfer from a mini-grid environment to Atari, but performance still falls short fromstate-of-the-art results on training Atari games from scratch.
[Figure]
We use the data-efficient version of RainbowDQN [20] as the base RL algorithm, i.e.we use the same training hyper-parameters andimage encoder network architecture as proposedin [35].
Note that RL 2 is excluded from thissection because it is on-policy.
5.1 Training Setup
Reptile-Rainbow incorporates Reptile [11]with the off-policy batch update in Rainbow.
Similar to Reptile-C2F-ARM, transitions fromeach Atari game are stored in a separate replaybuffer.
For multiple steps in the inner loop, theagent draws a batch from a randomly sampledbuffer and performs updates to the distributionalvalue network.
In the outer loop, the value network gets a soft update to mix parameters beforeand after the inner loop updates.
PEARL-Rainbow conditions a context embedding to the value network encoded
MT-Rainbow jointly trains RainbowDQN on all training games/tasks.
In contrast to Reptile-Rainbow,the replay batch in both MT-Rainbow and PEARL-Rainbow contains samples from multiple taskreplay buffers.
During each replay update, we randomly select a fixed number of tasks (less or equalto the total number of available training tasks), then draw an equal number of samples for each taskand construct the replay batch.
Note that, Atari games generally have different action dimensions, hence for all pretrained agents,the action space is padded to be the maximum possible size (discrete, 18-dimensional).
The extradimensions are all mapped to “No-op” for the games with smaller original action space.
For trainingfrom scratch, the agents are trained on the original action dimensions on their corresponding testgame.
5.2 Test-time Adaptation Setup
We use 5 test-time tasks, namely Pong, Assault, Battle Zone, Ms Pacman, and Beam Rider.
For
Reptile-Rainbow and MT-Rainbow, a trained agent is finetuned with Rainbow for 100,000 environment steps on each unseen game, following the data-efficient benchmark proposed in [35].
For
PEARL-Rainbow, the agent is rolled-out on a new game for 100,000 environment steps, and onlyupdates the context embedding by encoding newly collected experiences.
For both Reptile-Rainbow and MT-Rainbow, all network parameters are finetuned.
We provideadditional results in Appendix E for loading only the convolutional encoder and re-initializing all
MLP layers during finetuning.
This setup was previously investigated in [33] at a smaller scale,i.e. training a Breakout agent and fine-tuning on Pong, but the network architecture and base RLalgorithms differ from ours.
5.3 Results
We report the test-time adaptation performance for each pretrained agent in Figure 8.
The results aredivided into 3 sets, based on the number of training tasks provided to each compared algorithm.
Foreach set of experiments, the same training from “scratch” baseline is used (colored in gray).
For 5-task- and 10-task- trained agents, we observe high variance across different test-time gamesdespite being finetuned from the same pretrained agent.
For example, finetuning the Reptile-Rainbowagent trained on 10 tasks provides a clear advantage over training from scratch on Assault, but fallsshort on Ms Pacman and Beam rider. [Figure]
Consistent with results on RLBench,
PEARL-Rainbow also fails to adapt to theunseen task despite continuously updatingits context embedding for a sufficientlylong period.
PEARL-Rainbow’s zero-shotperformance on some of the games isclearly better than random (i.e. the firstdata point for training from scratch)— however, this benefit comes more from the diversity of training tasks than from the metatraining algorithm itself, which is evidentfrom observing that: (1) PEARL-Rainbowhas similar zero-shot performance to thatof MT-Rainbow and Reptile-Rainbow, and(2) as the number of training task goes from5 to 10, PEARL-Rainbow’s performanceslightly improves.
Overall, training fromscratch has competitive performance across all test-time tasks, which is in contrast to previousresults on Procgen and RLBench. Previous work has also found that transfer between Atari games ischallenging [36, 32, 37], and the common hypothesis is that the visuals and control strategies varytoo much across games for positive transfer. We provide further empirical results in Appendix E togain more insights
Related Work
Meta-Reinforcement Learning Meta-RL aims to find the best learning strategy that enablesfast adaptation to a new task via reinforcement learning.
This often relies on meta-training with adistribution of tasks and exploiting their shared structures.
Two main approaches include contextbased methods and gradient-based methods.
Context-based methods are trained to use recent rolloutexperiences from a new task to form a context that can be used to distinguish what task the policyis solving.
Previously, this context has been formed implicitly via an LSTM [16, 38], or explicitly,by passing trajectories through a separate encoder, whose output is given to a context-conditionedpolicy [12, 39].
Gradient-based methods perform test-time optimisation of hyperparameters [40],loss functions [41, 42], or network parameters [3, 43, 44].
The meta-RL approaches above have only been studied in fully observable state settings withshaped rewards; neglecting more realistic real-world scenarios, where rewards are often sparse, andobservations are high-dimensional (e.g. images, point clouds, etc).
There is limited work that studythese issues: for example, hindsight relabeling is used to aid in sparse reward setups e.g. [45], but usesfully observable states. Other approaches to sparse reward and partial observability include HyperX[46], DREAM [47], and MetaCure [48].
Out-of-distribution variation adaptation within the sametask is another challenging setup, where recent methods include model-identification, experiencerelabeling, [49], and adding symmetries [50]. Beyond context-based and gradient-based methods built on model-free RL algorithms, other lines ofwork include: model-based meta-RL , via meta-training a dynamics model and has seen successin enabling adapting to different hardware or terrain conditions on a legged millirobot [51]; metaimitation learning , has been applied to vision-based robot manipulation [10, 52, 53]; meta-learn RL algorithms , which aims to discover RL objectives or update rules that can be transferred acrossdifferent task environments [34, 54, 55, 56, 57]. Although in our experiments, we follow the original designs pf PEARL and RL 2 which don’t allowgradient updates during test time, recent work [58] has looked into the theoretical limitations ofcontext-based meta-RL algorithms in out-of-distribution variation adaptation setting [58], and shownthat adding gradient updates (i.e. finetuning) at test time helps improve adaptation.
The pretraining procedure in our experiments is multi-taskreinforcement learning (MTRL), where the training objective is simply finding a single best policyacross multiple tasks.
The main challenge in multi-task learning in general lies in multi-objectiveoptimization, and has been investigated in MTRL [59, 60] and applied to robotics [61].
Recently, Kurin et al.
[62] demonstrated that joint training with proper regularization achieves competitiveperformance with the more complicated multi-task algorithms.
This observation aligns with ourmulti-task training results
. 396 Meta- v.s. Multi-task pretraining in RL
Multi-variation pretraining followed by fine-tuning, alsocalled domain random search (DRS), is also shown to achieve comparable performance to meta-RLon existing state-based benchmarks [63].
Our work expands on this setup by training on more distincttasks instead of variations, and excluding the test-time task from training. Notably, the meta-learningsuite (e.g. ML10, ML45) in the MetaWorld [64] benchmark also poses such task generalizationchallenges, and finetuning is recently shown to be better than meta-RL algorithms such as RL 2 and MAML [24].
Conclusion
We perform a large-scale study on vision-based meta-RL across a truly diverse set of tasks.
Our resultsshow that when trained and tested on truly diverse reinforcement learning tasks, simple pretrainingand fine-tuning can perform equally as well as well as more complex meta-learning approaches. Thisis consistent with the findings within the computer vision community [4, 5, 6, 7], but in contrastto the large body of current literature, which shows that meta-RL is effective when evaluated onvariations of the same train-time task.
Our work is an initial step towards understanding the subtleties between meta-RL and multi-taskpretraining, with plenty of room for further investigation.
Despite the breadth of our experimentsacross 3 benchmarks, a limitation of our study is the limited number of pre-training tasks for both
RLBench and Atari.
This may not be sufficient to fully learn the representations required to enablethe best adaptation performance on new tasks.
However, note that the purpose of this study was notto show that multi-task pretraining followed by fine-tuning is better than training from scratch, butrather that multi-task pretraining can perform equally as well, or better, than meta-RL.
A Detailed justifications for the checklist questions
1.(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions andscope?[Yes
We believe our experiments provide an extensive study to our problem statement, comparingmulti-task pretraining and finetuning with meta-RL. The empirical results are comprehensive and clearlysupports our stated contributions.
(b) Did you describe the limitations of your work?
[Yes] Yes, see section 7(c)
Did you discuss any potential negative societal impacts of your work?[Yes]
Yes, see section 7(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?[Yes]
If you are including theoretical results...
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (eitherin the supplemental material or as a URL)?
[Yes] Please refer to the code scripts provided in thesupplemental material.(b)
Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
[Yes]Please refer to both main text and appendix for experiment details.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multipletimes)?[Yes]
All adaptation experiments in Procgen and RLBench are run for 3 seeds. In Atarim all adaptationexperiments are run for 10 seeds due to the high variance during RL training. (d)
Did
you include the total amount of compute and the type of resources used (e.g., type of GPUs, internalcluster, or cloud provider)
Yes As stated in section 2, we use RTX A5000 GPUs each with 24GB memory. Procgen experiments are alldone on single-GPU: it took 10 hours for training each multi-task agent for 100M environment steps,and 10 minutes for each finetuning/adaptation run on each of the 20 test levels. RLBench
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]We use the Procgen RL environment released by OpenAI under MIT lisence [13]; the RL training codeis built on the open-source RL framework implementation from Stable-baselines3 [65] We use the RLBench simulated robotic manipulation environment [8] released under MIT license. Our C2F-ARM algorithm and training framework are built based on the original author’s implementationand open-sourced code under MIT license.
We use The Arcade Learning Environment [15, 14] (under MIT license) for simulated RL environments
in our Atari experiments.
The code for RainbowDQN is built on the open-source implementation [66]
(b) Did you mention the license of the assets?
[Yes] See described above, all used assets are released under MIT
license. (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] Yes, please see the supplemental material for all the code for reproducing our experiments.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or
offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
[N/A] B Summary of Compared Algorithms In this section, we illustrate a unified overview of how the meta-RL
Require:
Learning rate α ; Reptile step size ϵ ; Policy optimization objective L ; Trajectory buffer D Input: Algorithm name algo Initialize policy π θ , LSTM embedder ϕ while not converged do if algo ==
Reptile then Sample a single task T ∼ p ( T ) Set θ old ← θ for i = 0 to k − 1 do Collect trajectories D i using π θi θ i +1 ← θ i − α ∇ θi L ( π θi , D i ) end for θ ← ϵ ( θ old − θ k ) else if algo ==
RL 2 then for i = 1 to b do Sample task T
i ∼ p ( T ) ; collect D
i using π θi ,ϕ end for θ ← θ i − α ∇ θ (cid:80) b i =1 L i ( π θ ,ϕ, D i )
ϕ
← θ i − α ∇ ϕ (cid:80) bi =1 L i ( π θ ,ϕ, D i ) else if algo ==
Pretrain then for i = 1 to b do Sample task T
i ∼ p ( T ) ; collect D
i using π θ end for θ ← θ − α ∇ θ (cid:80) bi =1 L ( π θ , D ) end if end while
Require:
Learning rate α ; Reptile step size ϵ Context sampling heuristics S c ; Value-function loss J Q ;
KL loss L KL ; Input: algorithm name algo
Initialize Q-function Q θ , context encoder ϕ ; while not converged do
Sample B replay buffers and draw samples from each buffer to construct batch b
= { b 1 ,...,b j } if s ==
Reptile then for j = 1 to B do θ oldj ← θ for i = 0 to k − 1 do θ i +1 ←
θ i − α ∇ θi J Q ( b j ) end for θ ← ϵ ( θ old − θ kj ) end for else if algo = = PEARL then for j = 1 to B do Sample contexts c j = S c ( D i ) L jKL =
L KL ( ϕ ( c j )) end for ϕ ← α ∇ ϕ (cid:80) L jKL θ ← θ − α ∇ θ J Q ( b, { c 1 ,...,c j } ) else if s ==
Pretrain then θ ← θ − α ∇ θ J Q ( b ) end if end while
C Additional Procgen Remarks
C.1 Experiment Details
[Figure]
C.2 Additional Results
How well does RL 2 adapt See Figure 9 fora comparison between the performance of the first 624 rollout and the second rollout at test time.
Results show that across all types of training task setups, the first trialhas similar rewards than the second trial, suggesting a failure to adapt.
D Additional RLBench RemarksD.1 Coarse-to-fine Attention Driven Robotic Manipulation (C2F-ARM)
A core component of C2F-ARM is the coarse-to-fine Q-attention [19] module, which takes as input a coarse 3D 629 voxelization of the scene, and learns to attend to interesting areas within the coarse voxelization.
The modulethen ‘zooms’ into that area and re-voxelizes the scene at a higher spatial resolution.
This ‘attend-and-zoom’ 631 procedure is applied iteratively and results in a near-lossless discretization of the 3D space.
This 3D space
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
discretization is combined with a rotation discretization to give a continuous next-best pose output.
C2F-ARM 633 takes this next-best pose and uses a motion planner to take the robot to the goal pose.
In this work, we use theoriginal C2F-ARM algorithm, and do not include any of its subsequent extensions, e.g., learned path ranking [67] 635 and tree expansion [68].
As mentioned above, a small number of demonstrations are used to overcome the exploration problem within 637 these sparse-reward environments.
The predecessor to C2F-ARM, ARM [31], introduced two demonstrationpre-processing procedures: (1) keyframe discovery , which assists the Q-attention at the initial phase of training by 639 suggesting meaningful points of interest; and (2) demo augmentation , which takes demo episodes and producesmany sub-episodes with different starting points, thereby increasing the initial number of demos in the replay 641 buffer.
All C2F-ARM results use both pre-processing procedures.
D.2 Single-task, Multi-variation Experiments
As a sanity check for whether the meta-RL algorithms are able to generalize to an easier, multi-variation setup, 644 we experiment with the multi-variation push_button task from RLBench, where the task is to push a button onthe tabletop but different variations differ in button colors.
Each method is trained on 10 variations and average 646 test-time performance on 5 unseen variations.
Results are reported in 11: all compared methods can perform anunseen variation in a zero-shot manner, and PEARL is able to adapt despite the lack of gradient updates.
D.3 Properties of fine-tuning Q-attention
tasks, when pretrained with multi-task pretraining.
In column “Success Rate (Train)”, we report the final training performance averaged across evaluating 30 episodes for each of the 10 training tasks.
In column “Success
Rate (Unseen Task)”, we report zero-shot direct evaluation performance of the trained agent on the held-out unseen task.
[Table]
Earlier in the paper, we have shown that multi- 650 task pre-training, followed by fine-tuning, canperform equally as well as meta-RL.
In the 652 following set of experiments, we explore finetuning in more detail, focusing on: (1) zero- 654 shot performance on test tasks (no test-timegradient updates); (2) investigate whether it is 656 better to fine-tune an unseen task in isolation,or together with other tasks (in a multi-task 658 setup); and (3 ) inspecting the role of each Qattention depth on fine-tuning performance.
We begin by evaluating zero-shot task perfor- 661 mance on held-out test tasks when pretrainedwith multi-task pretraining and evaluated on 30 663 episode rollouts.
Results in Table 2 show thatmulti-task pretraining, even on a small number 665 of tasks, can allow the Q-attention to begin tolearn an intuition of ‘objectness’, which can 667 be useful for zero-shot performance on sometasks.
The next set of experiments aims to investigate whether it is better to fine-tune an unseen task in isolation, or 670 together with other tasks (in a multi-task setup).
The intuition for the former is that train-time tasks (where wehave access to demos), can be used to learn good representations and exploration strategies; while the latter 672 intuition is that mixing with train-task data can act as auxiliary tasks, and the test-time task is treated as the maintask.
As shown in Figure 12, fine-tuning in isolation is superior to training in a multi-task setting.
The hypothesis 674 here is that the agent can keep the representations and skills that are useful to the fine-tune task, while forgettingnon-useful ones; whereas training with other tasks requires that the network have the capacity to remember all 676 skills.
The final set of experiments aim to inspect the role of each Q-attention depth on fine-tuning performance.
678 Figure 13 shows a comparison of three different fine-tuning strategies: (1) updating only the first (coarse)
Q-attention depth, (2) updating only the second (fine) depth, and (3) updating both depths. Unsurprisingly both the however depth almost equally during Q attention learns ‘objectness’ e
, what object to interact with “fine more concerned how interact object specific therefore [Figure] [Figure]
E Additional Atari RemarksE.1 Full adaptation results
See Figure 14 for separate adaptation results on all 5 test games.
For each training method (either training 688 from scratch, adaptation or finetuning), we run 10 seeds on every test game.
Because we use the data-efficientbenchmark [35] which does not train RainbowDQN to full coverage, we normalize the scores on each game by 690 the corresponding final RainbowDQN result, as reported in [20].
We then plot the normalized reward averagedover all 10 seeds.
The first and second row shows adaptation performance from two different training sets: 692 meta-training or pretraining using 5 tasks (top row) versus 10 tasks (bottom row).
Hyperparameters for training base RainbowDQN (data-efficient version) are shared across all compared algo- 694 rithms, see Table 3 for details.
Our code is built on the popular open-source implementation from [66].
E.2 Additinal Finetuning Experiments
We compare two ways of finetuning a multi-task agent: finetune all network parameters, versus finetune only 697 the convolutional layers and re-initialize the MLP layers.
Results are reported in Figure 15 We note again thesmall difference between the two sets of results and between finetuning and training from scratch, which further 699 suggests the possibility that
[Table]