L INEAR R EPRESENTATION M ETA -R EINFORCEMENT L EARNING FOR I NSTANT A DAPTATION
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper introduces Fast Linearized Adaptive Policy (FLAP), a new metareinforcement learning (meta-RL) method that is able to extrapolate well to outof-distribution tasks without the need to reuse data from training, and adapt almost instantaneously with the need of only a few samples during testing.
FLAP builds upon the idea of learning a shared linear representation of the policy so that when adapting to a new task, it sufﬁces to predict a set of linear weights.
A separate adapter network is trained simultaneously with the policy such that during adaptation, we can directly use the adapter network to predict these linear weights instead of updating a meta-policy via gradient descent, such as in prior meta-RL methods like MAML, to obtain the new policy.
The application of the separate feed-forward network not only speeds up the adaptation run-time signiﬁcantly, but also generalizes extremely well to very different tasks that prior MetaRL methods fail to generalize to.
Experiments on standard continuous-control meta-RL benchmarks show FLAP presenting signiﬁcantly stronger performance on out-of-distribution tasks with up to double the average return and up to 8X faster adaptation run-time speeds when compared to prior methods.
I NTRODUCTION
Deep Reinforcement Learning (DRL) has led to recent advancements that allow autonomous agents to solve complex tasks in a wide range of ﬁelds (Schulman et al.
(2015), Lillicrap et al. (2015a),
Levine et al.
However, traditional approaches in DRL learn a separate policy for each unique task, requiring large amounts of samples.
Meta-Reinforcement learning (meta-RL) algorithms provide a solution by teaching agents to implicitly learn a shared structure among a batch of training tasks so that the policy for unseen similar tasks can quickly be acquired (Finn et al.
Recent progress in meta-RL has shown efforts being made in improving the sample complexity of meta-RL algorithms (Rakelly et al.
(2019), Rothfuss et al.
(2018)), along with the out-ofdistribution performance of meta-RL algorithms during adaptation (Fakoor et al.
; Mendonca et al.
However, most of the existing meta-RL algorithms prioritize sample efﬁciency at the sacriﬁce of computational complexity in adaptation, making them infeasible to adapt to fastchanging environments in real-world applications such as robotics. In this paper, we present Fast Linearized Adaptive Policy (FLAP), an off-policy meta-RL method with great generalization ability and fast adaptation speeds.
FLAP is built on the assumption that similar tasks share a common linear (or low-dimensional) structure in the representation of the agent’s policy, which is usually parameterized by a neural network.
During training, we learn the shared linear structure among different tasks using an actor-critic algorithm.
A separate adapter net is also trained as a supervised learning problem to learn the weights of the output layer for each unique train task given by the environment interactions from the agent.
Then when adapting to a new task, we ﬁx the learned linear representation (shared model layers) and predict the weights for the new task using the trained adapter network.
An illustration of our approach is highlighted in Figure 2.
We highlight our main contributions below:
• State of the Art Performance.
We propose an algorithm based on learning and predicting the shared linear structures within policies, which gives the strongest results among the meta-RL algorithms and the fastest adaptation speeds.
FLAP is the state of the art in all these areas including performance, run-time, and memory usage. As is shown in Fig- ure 1, the FLAP algorithm outperforms most of the existing meta-RL algorithms including MAML (Finn et al.
, PEARL (Rakelly et al.
and MIER (Mendonca et al.
in terms of both adaptation speed and average return.
Further results from our experiments show that FLAP acquires adapted policies that perform much better on out-ofdistribution tasks at a rapid run-time adaptation rate up to 8X faster than prior methods.
• Prediction rather than optimization.
We showcase a successful use of prediction via adapter network rather than optimization with gradient steps (Finn et al.
or the use of context encoders (Rakelly et al.
during adaptation.
This ensures that different tasks would have policies that are different from each other, which boosts the out-ofdistribution performance, while gradient-based and context-based methods tend to produce similar policies for all new tasks. Furthermore, the adapter network learns an efﬁcient way of exploration such that during adaptation, only a few samples are needed to acquire the new policy.
To our knowledge, this is the ﬁrst meta-RL method that directly learns and predicts a (linearly) shared structure successfully in adapting to new tasks.
1. R ELATED WORK
The idea behind learning a shared linear structure is motivated by the great progress of representation learning and function approximation in domains such as robotics (Kober et al.
and natural language processing (Keneshloo et al.
Representation learning has seen success in cases of transfer learning RL in different tasks, and multi-agent reinforcement learning (MARL) where different agents may share common rewards or dynamics (Taylor & Stone (2009), Silva et al. The idea of learning shared information (embeddings) across different tasks has been investigated deeply in transfer learning, including for example, universal value function approximators (Schaul et al. , successor features (Barreto et al.
(2017), Hunt et al. (2019)), and invariant feature spaces (Gupta et al.
Using a prediction model during meta-testing presents a new idea into meta-RL.
The current trend in meta-reinforcement learning has been to either use an encoder based method through the use of context captured in recurrent models (Fakoor et al.
or variational inference (Rakelly et al.
(2019)), or use a variation of model-agnostic meta-learning (MAML)
(Finn et al.
(2017)), which aims at ﬁnding some common parametrization of the policy such that it sufﬁces to take one gradient step in adaption to acquire a good policy.
In contrast, FLAP aims at predicting the weights for the policy network directly in adaptation without the need of taking gradient steps or using context and comparing with existing data.
This gives better performance in (a) faster adaptation speeds due to the minimal amount of computation required; (b) smaller sample complexity during adaptation due to the better exploration strategy provided by adapter network; (c) strong out-of-distribution generalization ability resulting from the fact that the policy for each new task can be completely different in the last layer (in contrast, the new policies produced by gradient-based algorithms like MAML only differ in a single gradient update, which makes them highly similar, hurting the generalization ability).
We believe the idea of predicting instead of ﬁne-tuning via gradient descent can shed light to broader questions in meta-learning.
B ACKGROUND
We consider the standard Markov Decision Process (MDP) reinforcement learning setting deﬁned by a tuple T =
( S , A , µ, µ 0 , r, T ) , where S is the state space,
A is the action space, µ ( s t +1 | s t , a t ) is the unknown transition distribution with µ 0 being the initial state distribution, r ( s t , a t ) is the reward function, and T is the episode length.
Given a task T , an agent’s goal is to learn a policy π : S
→ A to maximize the expected (discounted) return:
[Equation]
We now formalize the meta-RL problem.
We are given a set of training tasks T train =
{T i } ni = and test tasks T test =
{T j } mj =1 .
In meta-training, we are allowed to sample in total k trajectories D ktrain from the tasks in T train and obtain some information φ ( D ktrain ) that could be utilized for adaptation.
The choice of φ ( D ktrain ) varies for different algorithms.
In meta-testing (or adaptation), we are allowed to sample l trajectories D ltest from each T i ∈ T test .
We would like to obtain a policy π ( φ ( D ktrain ) , D ltest ) for each testing task
T i , such that the average expected return for all testing tasks is maximized:
[Equation]
Here the policy only depends on the training data through the information φ ( D ktrain ) , and can be different for different test tasks.
We are interested in getting the expected return in equation 2 as large as possible with the smallest sample complexity k, l .
Furthermore, we hope that the information φ ( D ktrain ) is reasonably small in size such that the memory needed for testing is also controlled.
We also state that in-distribution tasks typically refer to the test tasks being drawn from the same distribution as the training distribution of tasks whereas out-of-distribution tasks represent the test distribution as very different from the training distribution (in our case, all out-of-distribution tasks are completely disjoint from the training set).
F AST L INEARIZED A DAPTIVE P OLICY (FLAP)
In this section, we describe the FLAP algorithm.
We ﬁrst assert and justify our assumption behind the goal of the FLAP algorithm (Section 3.1), and then discuss the high of the algorithm (Section 3.2).
3. A SSUMPTION AND M OTIVATION
Our FLAP algorithm is based on the following assumption on the shared structure between different tasks during training and testing:
Assumption 3.1 We assume that there exists some function φ : S → R d 1 such that for any task in the meta-RL problem T i ∈ {T train ∪ T test } , there exists a policy π i of the following form that maximizes the expected return q γ ( π, T i ) :
[Equation]
Here w i ∈ R d 1 × d 2 , b i ∈ R d 2 .
d 1 is the latent space dimension and d 2 is the dimension of the action space.
The function φ represents a shared structure among all tasks, and w i and b i are the unique weight and bias for task T i .
The function σ is the activation function depending on the choice of the action space.
Similar assumptions that constrain the policy space also appear in literature, e.g. Levine et al.
Lillicrap et al.
(2015b); Schmidhuber (2019)
; Mou et al. and references therein. One can also extend the linear representation in equation 3 to quadratic or higher-order representations for more representational ability and looser asumptions.
We compare the performance between linear structures and non-linear structures in Section C.1.3.
3. A LGORITHM OVERVIEW
Given assumption 3.1, the overall workﬂow of the algorithm is quite straightforward: in metatraining, we would like to learn the shared feature φ from the training tasks T train ; in meta-testing, we utilize the learned linear feature φ to derive the parameters w i , b i for each new task.
To speed up the adaptation, we also train a separate feed-forward adapter network simultaneously that takes the trajectories of each task as input and predicts the weights that are learned during training.
The full pseudo-code for both meta-training and meta-testing is provided in Appendix A.
3.2. M ETA - TRAINING
In order to learn the linear representation φ from Assumption 3.1, we use a deep neural network for the policy, sharing all but the output layer among all the train tasks in the training set.
Each train task will have a unique output layer during meta-training, and the shared neural network is the feature φ
we hope to learn during training.
Mathematically, we parametrize the policy via a neural network as π ( φ, w i , b i ) =
σ ( (cid:104) φ, w i (cid:105) + b i ) , where w i and b i are the weights and bias unique to task i , and σ represents the linear activation for the output layer since for all the tasks we considered in this paper, the action space takes values in the range of [ −∞ , + ∞ ] .
FLAP uses the general multi-task objective to train the policy over all the n training tasks during meta-training.
We maximize the average returns over all training tasks by learning the policy parameters represented by { φ, { w i , b i } ni =1 } .
We would like to ﬁnd the meta-learned parameters via solving the following optimization problem:
[Equation]
Here φ is parameterized by a neural network, and π i is parameterized by the parameters { φ, { w i , b i } ni =1 } .
Actor-Critic.
For solving the optimization problem in equation 4, we apply the off-policy soft actor-critic (SAC) (Haarnoja et al.
(2018)) for sample efﬁciency.
In the actor-critic model, besides the policy network, we also have a critic network which learns a
Q function for evaluating the current policy.
We linearize the critic network in a similar fashion to the policy by ﬁxing a set of shared layers in the critic network.
We show the shared layer structure for both policy and critic network in Figure 3.
Linearizing the critic network, though not explicitly required, does not incur any loss in performance from experimental results, and drastically reduces the number of parameters needed to be learned during training for the critic portion of the actor-critic method.
We discuss in more detail about the use of the soft actor-critic in Appendix D.1.
Adapter network.
Learning the shared structure is a standard approach in multi-task transfer learning (Pan & Yang (2009);
Asawa et al.
(2017); D’Eramo et al.
The key contribution and distinction of our algorithm is the use of an adapter network for rapid adaptation.
We train a separate adapter network simultaneously with the shared structure during meta-training.
We deﬁne the SARS tuple as a tuple containing information from one time step taken on the environment by an agent given a single task, including the current state, action taken, reward received from taking the action given the current state, and the next state the agent enters ( s t , a t , s t +1 , r ( s t , a t ) ).
The adapter net is fed a single tuple or a sequence of the most recent SARS tuples from a task and predicts the unique weights ( ˆ w i ) needed to acquire the new policy for that task.
We train the adapter network with mean squared error between the output of the adapter net ( ˆ w i ) obtained from the input SARS tuples, and the set of weights obtained from meta-training contained in each output layer.
The process of training the adapter net is shown in Figure 4.
[Figure]
of the adapter model.
3.2. M ETA - TESTING
In meta-testing, the FLAP algorithm adapts the meta-learned policy parameters w i , b i to a new task ( T i ∈ T test ) with very little data sampled from the new task.
Following Assumption 3.1, we assume that the shared structure φ is already learned well during meta-training.
Thus in meta-testing, we ﬁx the structure φ obtained in training, and only adjust the weights for each test task.
Mathematically, we run the following optimization problem to obtain the policy for each new test task T new ∈ T test :
The objective for the new task ( T new ) is to ﬁnd the set of weights that with the shared structure φ form the adapted policy
[Equation]
We then output π ( φ, w new , b new ) as the policy for the new task.
In our practical implementation and results, we show our contribution that requiring an objective function to be optimized during meta-testing does not necessarily have to be the only approach
[Figure]
for meta-RL, though FLAP can still have the optimization in Equation 5 easily integrated in metatesting.
FLAP performs rapid online adaptation by using the adapter network to predict the weights of the output layer as opposed to performing the optimization step.
The ﬁrst time-step taken by the agent is based off the most recent weights learned from the adapter network.
This then initiates a feed-back loop where now each time-step gives the adapter network a single tuple or a sequence of the most recent SARS tuples which inputted to the adapter network, immediately produces a set of predicted weights to use with the linear feature mapping ( φ ) to acquire an adapted policy ( π adapt = φ T · w i + b i ).
Only a ﬁxed number of online adaptation steps are needed to acquire the fully adapted policy.
Once the last time-step of weights is predicted, FLAP keeps only the last set of predictive weights for the rest of the episode.
The meta-testing loop is shown in Figure 5.
E XPERIMENTS
This section presents the main experients of our method.
We evaluate FLAP in Section 4.1 by comparing it to prior meta-learning approaches on in-distribution tasks.
We then examine how the FLAP algorithm is able to generalize to very different test tasks using out-of-distribution tasks in Section 4.2.
We take a holistic view of FLAP as a meta-RL method and analyze its run-time and memory usage on an autonomous agent during the meta-testing stage in Section 4.3. We leave the ablation studies and sparse reward experiments to Appendix C.
Finally, we provide thorough discussions on the empirical performance in Section 4.4.
4. S AMPLE E FFICIENCY AND P ERFORMANCE C OMPARED TO P RIOR M ETHODS
We evaluate FLAP on standard meta-RL benchmarks that use in-distribution test tasks (training and testing distributions are shared).
These continuous control environments are simulated via the
MuJoCo simulator (Todorov et al.
and
OpenAI Gym (Brockman et al.
Rakelly et al.
(2019) constructed a ﬁxed set of meta-training tasks ( T train ) and a validation set of tasks ( T test ).
These tasks have different rewards and randomized system dynamics (Walker-2D-Params).
To enable comparison with these past methods, we follow the evaluation code published by Rakelly et al.
and follow the exact same evaluation protocol.
In all the tasks we consider, the adapter network is fed with a single SARS tuple during training and adaptation. Other hyper-paramters of our method are given in Appendix {+D.
+} and we report the result of FLAP on each environment from 3 random seeds.
We compare against standard baseline algorithms of MAML (Finn et al.
, RL2 (Duan et al. (2016b)), ProMP (Rothfuss et al.
, PEARL (Rakelly et al.
MQL (Fakoor et al.
(2019)), and MIER (and MIER-wR) (Mendonca et al.
We obtained the training curves for the ﬁrst four algorithms from Rakelly et al.
(2019) and the latter two from Mendonca et al.
4. G ENERALIZATION TO O UT - OF -D ISTRIBUTION T ASKS
To evaluate generalization we test on the extrapolated out-of-distribution (OOD) environments, where the train and test distributions are highly dissimilar and completely disjoint, created by Mendonca et al.
(2020) and Fakoor et al.
(2019), closely following their evaluation protocols.
The environments include changes in rewards and dynamics.
We run FLAP on 3 random seeds to evaluate the performance.
In all the tasks we consider, the adapter network is fed with a single SARS tuple during training and adaptation.
Hyperparameters are listed in Appendix D.3.
4. O VERALL M ODEL A NALYSIS
In this section, we assess the run-time of FLAP.
We compare our method to MIER-wr and PEARL in meta-testing speeds on benchmarks from both in-distribution and out-of-distribution tasks.
MIER and MQL both reuse the training buffer during testing, causing adaptation speeds and memory usage that are far worse than prior methods (MIER took 500+ seconds for meta-testing).
We only compare with the most competitive algorithms speed and memory-wise to showcase FLAP’s strength.
We defer the analysis of storage efﬁciency to Appendix (B).
To calculate the run-time speeds, we use the Python Gtimer package and a NVIDIA GEFORCE GTX 1080 GPU and time each of the Meta-RL methods on four MuJoCo environments (Ant, Cheetah,
Humanoid, Walker) on 30 validation tasks after allowing each algorithm to achieve convergence during meta-training. 4. D ISCUSSION
Beyond the experiments on in-distribution and out-of-distribution tasks, we did more experiments and analysis to justify the superiority of FLAP, including ablation studies and extra experiments on
MDPs with sparse rewards. We brieﬂy describe the results and take-aways in Section 4.4.1 and 4.4.2, and leave the details on experiments to Appendix C. Lastly, we analyze all the experimental results and summarize the reason for the validity and generalization ability of the adapter network in Section 4.4. 4.4. A BLATION STUDIES We validate the effectiveness of FLAP via a large amount of ablation studies.
In particular, we compare the performance under the following different settings:
With adapter net vs Without adapter net: One may wonder whether the adapter network really boosts the performance compared to the approach of directly learning the linear weights via soft actor-critic during adaptation without the help of the adapter network.
As is shown in Appendix C.1.1, although both algorithms converge to approximately the same average reward in the end, the adapter network drastically decreases the sample complexity during the adaptation process and accelerates the speed (at least 30X faster among all the experiments) compared to the algorithm without the adapter.
Shared linear layer vs Non-linear layers: One may wonder why we assume the policies share only the very last layer instead of the last two or more layers as embeddings.
Indeed, experiments in Appendix C.1.3 show that when sharing two (or more) layers, the policies may have too much degree of freedom to converge to the global minimum.
The shared linear layer turns out to be the best assumption that balances the performance with the speed. Single SARS tuple vs a Sequence of SARS tuple as Input of Adapter Net: All the experiments above work with the adapter net with a single (the most recent) SARS tuple as input. We show in Appendix C.1.4 that if we let the adapter net take a sequence of
the most recent SARS tuples as input, the average return would not be affected on the tasks we considered above. However, we see a reduction in variance with a slightly slower adaptation speed. Overall, taking a longer sequence of SARS tuples as input tends to stabilize the adaptation and reduce the variance in the sacriﬁce of runtime. However, for some speciﬁc tasks with sparse rewards and similar SARS tuples, it also helps improve the average return, as is discussed in the next section.
We show in Appendix C.2 that for tasks with sparse rewards and similar dynamics (e.g. the navigation tasks), the performance of FLAP may deteriorate a little since the resulting policy highly relies on the chosen transition tuple, which may lead to higher variance. However, it is still comparable to PEARL if we let the adapter net take a sequence of SARS tuples as input.
This is due to fact that a sequence of SARS tuples provides more information on the context for the adapter, making the resulting policy less dependent on the single chosen transition tuple.
However, for most of the other tasks we considered above, one single transition tuple sufﬁces to provide enough information for the success of adapter network with small variance and strong generalization ability.
We also validate it in Appendix C.3 by showing that the loss for adapter net converges in a reliably fast and stable way.
4.4. O N THE VALIDITY AND GENERALIZATION ABILITY OF FLAP AND THE A DAPTER N ETWORK
To summarize the experimental results above, we see that FLAP guarantees a much smaller sample complexity during adaptation, improved generalization ability and faster adaptation speeds. We provide justiﬁcation as to why FLAP can achieve all the properties separately. Small Sample Complexity: From our ablation study in Appendix C.1.1, it is clear that the small
sample complexity during meta-testing comes from the introduction of adapter net. We conjecture that during training, the adapter net learns the structure of
optimal policies shared by all the training tasks and during adaptation, the predicted weights encourage the most
efﬁcient exploration over potential structures of optimal policies such that the average return converges in a fast and stable way. Strong generalization: From our ablation study in Appendix C.1.1, we see that the algorithms with and without adapter converge to a similar average return. Thus we conclude that the generalization ability is not related to the speciﬁc design of adapter net, but comes from the shared linear structure. In traditional gradient-based meta-RL approaches like MAML, the new policies are generated from taking one gradient step from the original weights in training, which makes the new policies very similar to the old ones.
In contrast, the new policy in FLAP can have a completely different last layer, which guarantees more ﬂexibility in varying policies, and thus a better generalization to new tasks. Fast adaptation: Compared to other meta-RL approaches, FLAP only requires evaluating the output of a ﬁxed adapter net, which provides instant adaptation when the adapter net is small enough. The successful experiments also justify the validity of Assumption 3.1: for all the tasks we consider, they do have some near-optimal policies with shared embedding layers. We believe this could also provide interesting theoretical directions as well.
C ONCLUSION
In this paper, we presented FLAP , a new method for meta-RL with signiﬁcantly faster adaptation run-time speeds and performance.
Our contribution in the FLAP algorithm presents a completely new view into meta reinforcement learning by applying linear function approximations and predicting the adaptation weights directly instead of taking gradient steps or using context encoders.
We also believe the idea behind the adapter network during meta-testing can be extended to other meta learning problems outside of reinforcement learning.
B M EMORY U SAGE
For storage analysis of models, we concern ourselves only with the amount of memory needed during meta-testing as the process of meta-learning is to meta-learn a policy that is both lightweight in storage, fast to adapt, and effective in performance on a new task.
We assessed the internal structure of models based off the number of parameters used in each algorithm for the models that are necessary when adapting to a new task.
As already noted in Section 4.3, we only use MIER without experience relabeling (Mendonca et al.
and PEARL (Rakelly et al.
in comparison with FLAP.
We note that though MIER with experience relabeling is better on the Ant-Negated Task in comparison with FLAP, it utilizes the process of experience relabeling where the training replay buffer must be used for meta-testing.
MQL (Fakoor et al.
(2019)) also requires the use of the training buffer for adapting to out-of-distribution tasks.
This leads to a highly negative result in both algorithms since the amount of memory necessary for these models to adapt to highly extrapolated tasks grows as the number of meta-train tasks used during training increases and the complexity of the meta-RL problem increases.
PEARL and MIER-wR (without experience relabeling) do not need to reuse the replay buffer however FLAP performs uniformly better than these methods in all the out-of-distribution baselines.
The model parameters are detailed in Table 1 and clearly FLAP is at least comparable if not better in the aspect of memory required for adaptation.
[Table]
C A BALATION S TUDIES AND F URTHER E XPERIMENTS
In this section, we showcase some more experiments involving FLAP.
C. E XPERIMENTAL D ETAILS ON ABLATION STUDY
We conclude our experimental analysis by showcasing ablation studies on the adapter network.
We ﬁrst provide further justiﬁcation for the use of the adapter network in the ﬁrst place and evidence that it works with reasonable decreasing loss while the model learns.
We then showcase the effect during adaptation of feeding in different input data into the adapter network.
We ﬁnally ﬁnish by showcasing single-point input compared to a sequence of points concatenated together as input to the adapter network and its result on the effectiveness of the adapter network.
Analysis of the loss values of the adapter network during training can be found in Appendix C.3.
C.1. W ITH ADAPTER NET VS WITHOUT ADAPTER NET : S TRONG P ERFORMANCE OF THE A DAPTER N ETWORK
To analyze the importance of the adapter network, we compare the adaptation (meta-test) results on the Half-Cheetah environment with the use of feed-forward prediction compared to simply using a meta-testing objective and performing SAC objective function gradient updates to iteratively acquire the new policy.
Although both methods are able to eventually obtain a strong performing policy, the adapter network drastically decreases the amount of samples needed for adaptation and provides a clear beneﬁt for the FLAP algorithm, as is shown in Figure C.1.1.
We also see a dramatic decrease in run-time (at least 30X among all the experiments) when we have the adapter network in the adaptation procedure.
C.1. SAS INPUT VS SARS INPUT
We test different inputs to the adapter network.
We train the adapter using only (state, action, next state) pairs compared to the original complete SARS pair to analyze whether the adapter is able to learn the reward function.
Given the random and poor performance of the adapter with only (S, A, S (cid:48) ) as input, we provide evidence that the adapter net learns valuable information to aid the algorithm during adaptation.
C.1. N ONLINEAR O UTPUT L AYERS
Our idea of maintaining a shared set of layers and a linear output layer can easily be extended beyond the linear case.
We motivate this section by showcasing that
FLAP does not gain performance from increased expressiveness when extended to the multi-layer output case.
We use a two-layer (quadratic) output and reduce the hidden units in each layer from 300, which was used in the linear case, to 50 to keep the total number of output units in both FLAP algorithms approximately consistent.
We also use the adapter network (learning non-linear weights during meta-testing from scratch slowed down adaptation speeds) to predict both layer weights.
We ran the regular and modiﬁed FLAP algorithms on out-of-distribution tasks to compare generalization strength of each method.
Our implementation of the modiﬁed FLAP algorithm is illustrated in Figure 11.
Results of the comparison are shown in Figure 12.
[Figure]
S EQUENCE VS . S INGLE P OINT OF SARS TUPLES AS INPUT OF ADAPTER NET
We showcase that using a sequence of tuples (5 tuples) does not improve the performance of the algorithm in our baseline problems and in cases where the linear assumption holds up well, although it offers improvements including reduced variance in less structured cases where the assumption may not hold as strongly such as in sparse reward settings (Appendix C.2).
It is important to note that the adapter is batch-wise trained meaning that a random batch of SARS tuples are sampled from the replay buffer of the corresponding task and all trained with the same set of target weights.
Although the network may be presented a tuple where the state or action is completely new, the adapter is able to generalize and make a strong prediction with just a single point and without sacriﬁce to performance indicating the self-correcting nature of the adapter network.
C. S PARSE R EWARD E NVIRONMENTS
Sparse Rewards Reinforcement Learning environments require the agent to reason about uncertainty in an environment where different tasks share many common features and seemingly appear identical.
We compare against the PEARL algorithm on the same (in-distribution) sparse reward setting (Sparse Point Robot) ﬁrst introduced by PEARL that requires navigation in a 2D setting to a goal target (Rakelly et al.
We mention that although FLAP only needs 1 trajectory of test data, we allow the PEARL algorithm to obtain as many trajectories from the unseen task as necessary to achieve its strongest performance which already requires 20+ trajectories even in such a simple problem.
We also run FLAP using a single point adapter input and a sequence of points (5 tuples) as an input to the adapter network.
In this case, the sequence of points as input helped not only improve the performance but also presented less variance among different runs of the algorithm which indicates that on problems where the linear assumption may not be as strong, such as sparse reward settings, using multiple points as inputs may present improved results.
Our algorithm completely outperforms PEARL when both algorithms are only allowed 1 trajectory of data, indicating that FLAP has better scalability especially when more complex sparse reward problems are encountered where more test data is required to be sampled to acquire the new policy.
In conducting the experiment, we followed the PEARL algorithm in training on dense rewards for the sparse point robot and testing on the sparse reward setting to allow for the fair comparison between the algorithms.
We state that the success of the FLAP algorithm may be attributed to the fact that the implementation is built on the SAC algorithm which uses entropy to help with exploration, a vital component especially in sparse reward problems.
C. A DAPTER N ETWORK L OSS
During training, we captured the loss of the adapter network during training for its ability to train on target train task weights.
In the experiments conducted, the loss value typically converged nicely overall with no indication of any high loss values which would have presented conﬂicting performance on the different baseline tasks.
We showcase the loss on the Half-Cheetah environment and the Sparse Point Robot Problem where there are sparse rewards.
Although the adapter network trained reasonably on both, it is important to note that when the linear assumption is not as strong, the loss value tends not to converge as smoothly which also leads to variance in the performance of the algorithm on the environment.
We provide implementation details including the hyper-parameters used in building FLAP and the settings used for evaluation of algorithms on both in-distribution and out-of-distribution tasks.
D. A CTOR -C RITIC I MPLEMENTATION
We discuss more speciﬁcs on the use of the actor-critic algorithm in our model.
Speciﬁcally, for the actor and critic networks, we parameterize the critic network by ψ , leading to Q ψ , and the policy network by θ , leading to π θ , then we can write the objective function for actor-critic algorithms to optimize the policy (actor) network:
[Equation]
where Equation 7 is the Q-value estimation from the critic and the policy loss J π in Equationspeciﬁcally details the loss function used to optimize the meta-policy parameters.
We also employ automatic entropy-tuning for our SAC algorithm to help with exploration and employ two
Q-function networks to help reduce over-estimation bias, a technique commonly known as ”doubleQ-learning” (van Hasselt et al.
D. E XPERIMENT D ETAILS
The horizon for all environments is 200 time-steps.
Evaluation of return is then the average reward of the trajectories where each trajectory is the sum of the rewards collected over its rollout.
We use settings from PEARL (Rakelly et al.
for the in-distribution tasks and the settings from
MIER (Mendonca et al.
for the out-of-distribution tasks.
The settings for the environments are listed in the Table.
FLAP requires less meta-train tasks for meta-training than prior methods.
The number of adaptation steps refers to the number of iterations of the feedback loop illustrated by meta-testing and described in Section 4 before the policy is ﬁnally settled.
The settings are detailed in Table 2 and Table
[Table]
[Table] [Table]
D. H YPER - PARAMETERS
The hyper-parameters for FLAP are kept mostly ﬁxed across the different tasks except for the number of parameter updates per iteration which is different only for the Ant and Walker environments
All others use the default of 1000 updates.
These hyper-parameters were kept as default values from the PEARL (Rakelly et al.
soft actor-critic implementation code in the PEARL code base as both FLAP and PEARL are built based off the rllab code base (Duan et al.
(2016a)).
They are detailed in Table
[Table] [Table]