Multi-Source Transfer Learning for Deep Model-Based Reinforcement Learning
Abstract
A crucial challenge in reinforcement learning is to reduce the number of interactions with the environment that an agent requires to master a given task.
Transfer learning proposes to address this issue by re-using knowledge from previously learned tasks.
However, determining which source task qualifies as optimal for knowledge extraction, as well as the choice regarding which algorithm components to transfer, represent severe obstacles to its application in reinforcement learning.
The goal of this paper is to alleviate these issues with modular multi-source transfer learning techniques.
Our proposed methodologies automatically learn how to extract useful information from source tasks, regardless of the difference in state-action space and reward function.
We support our claims with extensive and challenging cross-domain experiments for visual control.
Introduction
Reinforcement learning (RL) agents typically interact with a single unknown environment in order to master a given task.
Generally, the number of interactions required to do so is enormous, limiting the applicability of RL to the real world.
Recent works on model-based RL (MBRL) address this issue of sample inefficiency by learning a world model, which allows an agent to simulate interactions with complex environments (Ha & Schmidhuber, 2018; Hafner et al., 2019; 2020).
However, learning an accurate model of the environment from scratch still requires a substantial amount of interactions.
A common approach for enhancing sample efficiency in the supervised learning domain is transfer learning (Yosinski et al., 2014; Sharif Razavian et al., 2014; Zhuang et al., 2020).
By re-using the parameters of a neural network that were fit to some previous task (referred to as source task) for learning some other task (referred to as target task), one can reduce the number of samples required substantially compared to fitting randomly initialized parameters.
However, the performance of transfer learning highly depends on the choice of the source task, which is often based on the intuition of the designer (Taylor & Stone, 2009b).
As RL environments can differ in several fundamental aspects, the application of transfer learning in this domain is especially challenging and has received little attention.
In this paper, we address this issue and propose to alleviate it with multi-source transfer learning techniques.
Rather than re-using information from a single source task, multi-source transfer learning proposes to extract knowledge from multiple source tasks.
Our proposed methodologies autonomously learn to extract useful information from a collection of source tasks, regardless of the differences between environments, alleviating the necessity of selecting an optimal source task.
We apply our proposed techniques to the state-of-the-art world model-based algorithm for visual continuous control tasks, Dreamer (Hafner et al., 2019).
We focus on world model-based algorithms, as learning reward, dynamics, policy, and value models in a compact latent space of a world model has shown promising results for the application of transfer learning, yet little research has been conducted in this direction (Zhu et al., 2020).
A major obstacle of applying transfer learning to these types of algorithms is that they are composed of several different components.
Each component may or may not benefit from different types of transfer learning, dependent on their objective function and the differences between source and target tasks.
We provide valuable insights and empirical evidence on the transferability of such components, by applying multi-source transfer learning to Dreamer in a modular fashion, in addition to introducing a novel type of transfer learning that allows the fractions of parameters to be transferred.
We extensively evaluate the proposed methods in challenging cross-domain transfer learning experiments.
We consider two multi-source transfer learning settings: a single agent that masters multiple tasks, and multiple individual agents that each master a single task.
We provide techniques that allow cross-domain transfer learning for both of these settings, which are also applicable in single-source transfer learning settings.
However, we encourage the usage of multi-source transfer learning, in order to avoid manual selection of an optimal source task.
The contributions are summarized as follows:
• Fractional transfer learning : We introduce a novel type of transfer learning that allows fractions of parameters to be transferred, as an alternative to discarding information by random initialization, resulting in substantial performance improvements.
• Modular multi-task transfer learning : We show that training a single world model-based agent on multiple tasks simultaneously and transferring its components in a modular fashion with different types of transfer learning, results in enhanced sample efficiency and performance compared to learning from scratch.
• Meta-model transfer learning : We propose a multi-source transfer learning approach that allows models from several individual agents trained in different environments to be combined and transferred in a shared feature space.
These components produce additional input signals for the corresponding model of the target agent, allowing us to construct a meta-model that learns to weigh the usefulness of these signals for learning the target task, resulting in significant performance gains when applied to merely one model.
Preliminaries
We formalize an RL problem as a Markov Decision Process (MDP) (Bellman, 1957), which is a tuple (  ,  , 𝑃, 𝑅) , where  denotes the state space,  the action space, 𝑃 the transition function, and 𝑅 the reward function.
For taking a given action 𝑎 ∈  in state 𝑠 ∈  , 𝑃(𝑠 ′ |𝑠, 𝑎) denotes the probability of transitioning into state 𝑠 ′ ∈  , and 𝑅(𝑟|𝑠, 𝑎) yields an immediate reward 𝑟 .
The state and action spaces define the domain of the MDP.
The objective of RL is to find an optimal policy 𝜋 ∗ ∶  →  that maximizes the expected cumulative reward.
The expected cumulative reward is defined as 𝐺 𝑡 =
𝔼
[∑ ∞𝑡=0 𝛾 𝑡 𝑅 𝑡 ] , where 𝛾 ∈ [0, 1) represents the discount factor, and 𝑡 the time-step.
The general concept of transfer learning aims to enhance the learning of some target task by re-using information obtained from learning some source task.
Multi-source transfer learning aims to enhance the learning of a target task by re-using information from a set of source tasks.
In RL, a task is formalized as an
MDP 1 𝑀 with some optimal policy 𝜋 ∗ .
As such, in multi-source transfer learning for RL we have a collection of 𝑁 source MDPs 
=
{(  𝑖 ,  𝑖 , 𝑃 𝑖 , 𝑅 𝑖 )} 𝑁𝑖=1 , each with some optimal policy 𝜋 ∗𝑖 .
Let 𝑀 =
(  ,  , 𝑃, 𝑅) denote some target MDP with optimal policy 𝜋 ∗ , where 𝑀 is different from all 𝑀 𝑖 ∈  with regards to  ,  , 𝑃, or 𝑅 Multi-source transfer learning for reinforcement learning aims to enhance the learning of 𝜋 ∗ by reusing information obtained from learning 𝜋 ∗1 , .., 𝜋 ∗𝑁 .
When 𝑁 = 1 we have single-source transfer learning.
In this paper, we use Dreamer as a reference for deep MBRL algorithms (Hafner et al., 2019).
MBRL algorithms learn to model 𝑃 and 𝑅 internally.
Dreamer learns to model these functions in a compact latent space learned from visual observations, referred to as a world model Ha & Schmidhuber (2018).
As such, interactions with environments with high-dimensional observations can be imagined in a computationally efficient manner, which is used to facilitate sample-efficient policy learning.
The policy is used to collect data from the environment, which is used for learning the world model.
The world model consists of four components: a representation model 𝑝 𝜃 (𝑠 𝑡 |𝑠 𝑡−1 , 𝑎 𝑡−1 , 𝑜 𝑡 ) , observation model 𝑞 𝜃 (𝑜 𝑡 |𝑠 𝑡 ) , reward model 𝑞 𝜃 (𝑟 𝑡 |𝑠 𝑡 ) , and transition model 𝑞 𝜃 (𝑠 𝑡 |𝑠 𝑡−1 , 𝑎 𝑡−1 ) , where 𝑝 denotes distributions that generate real environment samples, 𝑞 denotes distributions approximating those distributions in latent space, and 𝜃 denotes the jointly optimized parameters of the models.
At a given timestep 𝑡 , the representation model maps a visual observation 𝑜 𝑡 together with the previous latent state 𝑠 𝑡−1 and previous action 𝑎 𝑡−1 to latent state 𝑠 𝑡 .
The transition model learns to predict 𝑠 𝑡 from 𝑠 𝑡−1 and 𝑎 𝑡−1 .
The reward model learns to predict the reward 𝑟 𝑡 corresponding to 𝑠 𝑡 .
The observation model reconstructs 𝑠 𝑡 to match 𝑜 𝑡 , providing the learning signal for learning the feature space.
This world model is called the recurrent state space model (RSSM), and we refer the reader to Hafner et al.
(2018) for further details.
In order to imagine a trajectory {(𝑠 𝜏 , 𝑎 𝜏 )} 𝑡+𝐻𝜏=𝑡 of length 𝐻 , where 𝜏 denotes the imagined time index, the representation model maps an initial observation 𝑜 𝑡 to latent state 𝑠 𝜏 , which is combined with action 𝑎 𝜏 yielded by the policy, to predict 𝑠 𝜏+1 using the transition model.
The reward model then predicts the corresponding reward 𝑟 𝜏+1 , which is used for policy learning.
In order to learn a policy, Dreamer makes use of an actor-critic approach, where the action model 𝑞 𝜙 (𝑎 𝜏 |𝑠 𝜏 ) implements the policy, and the value model 𝑣 𝜓 (𝑠 𝑡 )
≈ 𝔼 𝑞(⋅|𝑠 𝜏 ) (
∑ 𝑡+𝐻𝜏=𝑡 𝛾 𝜏−𝑡 𝑟 𝜏 ) estimates the expected reward that the action model achieves from a given state 𝑠 𝜏 .
Here 𝜙 and 𝜓 are neural network parameters for the action and value model respectively, and 𝛾 is the discount factor.
The reward, value, and actor models are implemented as Gaussian distributions parameterized by feed-forward neural networks.
The transition model is a Gaussian distribution parameterized by a Gated Recurrent United (GRU; Bahdanau et al.
followed by feed-forward layers.
The representation model is a variational encoder (Kingma & Welling, 2013; Rezende et al., 2014) combined with the GRU, followed by feed-forward layers.
The observation model is a transposed convolutional neural network (CNN; LeCun et al.
Related Work
In supervised learning, parameters of a neural network are transferred by either freezing or retraining the parameters of the feature extraction layers, and by randomly initializing the parameters of the output layer to allow adaptation to the new task (Yosinski et al., 2014; Sabatelli et al., 2018; Cao et al., 2021).
Similarly, we can transfer policy, value, and reward models, depending on the differences of the state-action spaces, and reward functions between environments (Carroll & Peterson, 2002; Schaal et al., 2004; Fern´andez & Veloso, 2006; Rusu et al., 2016; Zhang et al., 2018).
We can also transfer autoencoders, trained to map observations to latent states (Chen et al., 2021).
Experience samples collected during the source task training process can also be transferred to enhance the learning of the target task (Lazaric et al., 2008; Tirinzoni et al., 2018).
The transferability of deep model-free RL algorithms doesn’t appear to be promising, as was recently shown by Sabatelli & Geurts (2021).
However, they are suitable for distillation techniques, where a new neural network learns to predict the mappings of inputs to outputs of a pre-trained network (Hinton et al., 2015; Parisotto et al., 2015; Rusu et al., 2015).
By sharing a distilled policy across multiple agents learning individual tasks, Teh et al.
(2017) obtain robust sample efficiency gains.
In order to address the major obstacle of optimal task selection, Ammar et al.
(2014) introduced an autonomous similarity measure for
MDPs based on restricted Boltzmann machines, though it assumes the MDPs are within the same domain.
Garc´ıa-Ram´ırez et al.
(2021) propose to select the best source models among multiple model-free models using a regressor.
This paper focuses on transfer learning for deep MBRL, where we also need to consider the dynamics model.
Recent works showed that the dynamics model can be fully transferred between different domains if the environments are sufficiently similar (Eysenbach et al., 2020; Rafailov et al., 2021).
Landolfi et al.
(2019) perform a multi-task experiment, where the dynamics model of an MBRL agent is transferred to several novel tasks sequentially, and show that this results in significant gains of sample efficiency.
PlaNet (Hafner et al., 2018) was used in a multi-task experiment, where the same agent was trained on tasks of six different domains simultaneously using a single world model (Ha & Schmidhuber, 2018).
Similarly, In
Plan2Explore (Sekar et al., 2020) the authors show that a single global world model can be trained taskagnostically, after which a Dreamer (Hafner et al., 2019) agent can use the model to zero-shot or few-shot learn a new task within the same domain.
Dreamer is not to be confused with DreamerV2 (Hafner et al., 2020), which is essentially the same algorithm adapted for discrete domains.
Unlike previous works, we investigate multi-source transfer learning with a focus on world model-based algorithms, by introducing multiple techniques that enhance sample efficiency without the need of selecting an optimal source task.
Methods
Here we present the main contributions of this work.
we propose multi-task learning as a multi-source transfer learning origin, combined with modular and fractional transfer learning.
We follow this with an alternative setting, where we propose to transfer components of multiple individual agents utilizing a shared feature space and meta-model
4.1 Multi-Task Transfer Learning
In this section, we describe how we train a single agent on multiple MDPs simultaneously, and introduce a novel type of transfer learning that allows fractions of parameters to be transferred.
We then discuss what type of transfer learning is most suitable for each component of the agent.
4.1.1 Simultaneous Multi-Task Learning
We propose to train single agents facing multiple unknown environments simultaneously, each with different stateaction spaces and reward functions, in order to use them as multi-source transfer learning origins.
We train the multitask RL agents in a similar fashion to the experiments done by Hafner et al.
(2018), where the action dimension of each task is padded with unused elements to the size of the task with the largest action dimension.
We are required to have uniform action dimensions across all source environments, as we use a single policy model.
The agent collects one episode of each task in collection phases in order to acquire a balanced amount of experiences in the replay buffer.
4.1.2 Fractional Transfer Learning
Typically, one can either fully transfer the parameters of a neural network layer, or randomly initialize the parameters.
Feature extraction layers of neural networks are generally fully transferred (Sermanet et al., 2013).
The output layer of a neural network is often randomly initialized, in order to prevent overfitting.
However, this means that we discard all information contained in the parameters of the output layer.
We propose a simple alternative: fractional transfer learning (FTL), which allows us to transfer fractions of parameters.
Rather than discarding all information, this technique allows us to transfer a portion of previously obtained information, without risking overfitting.
Let 𝜃 𝑇 denote target parameters, 𝜃 𝜖 randomly initialized weights, 𝛼 the fraction parameter, and 𝜃 𝑆 source parameters, then we can do FTL by 𝜃 𝑇 =
𝜃 𝜖 +𝛼𝜃 𝑆 .
That is, we add a fraction of the source parameters to the randomly initialized weights.
4.1.3 Modular Transfer learning
Modern deep MBRL algorithms often consist of several components, each relating to different elements of an MDP.
Therefore, in order to transfer the parameters of such architectures, we need to consider transfer learning on a modular level.
Using Dreamer as a reference, we discuss what type of transfer learning each component benefits most from.
We consider three types of transfer learning: random initialization,
FTL, and full transfer learning.
We found in initial experiments that fully transferring all feature extraction layers of each component results in the best learning enhancements, and found that the output layer mainly impacts the transfer learning performance.
Therefore, we only discuss transferring parameters of output layers of the multi-task agents.
First, we consider the action model, which implements the policy.
As action elements don’t match across different environments, our initial experiments showed, as anticipated, that both full transfer and FTL across environments with different action spaces results in detrimental performance drops compared to random initialization.
Next, we found that fractionally transferring parameters of the reward and value models can result in substantial performance gains (Appendix E).
In this paper, we apply our methods to MDPs that have similar reward functions, meaning the parameters of these models consist of transferable information that enhances the learning of a target task.
This demonstrates the major benefits of FTL, as the experiments also showed that fully transferring these parameters has a detrimental effect on learning (Appendix D).
Finally, initial experiments showed that fully transferring the parameters of the representation, observation, and transition model results in the best performance improvements for most environments.
As we are dealing with visually similar environments, the generality of convolutional features allows full transfer of the representation and observation parameters (Chen et al., 2021).
Additionally, when environments share similar physical laws (e.g. a physics engine) transition models can often be fully transferred, provided that we reset the weights connected to actions (Taylor & Stone, 2009a; Landolfi et al., 2019).
4.2 Multiple-Agent Transfer Learning
In this section, we consider an alternative multi-source transfer learning setting, where we have access to the parameters of multiple individual agents that have learned the optimal policy for different tasks.
Transferring components from agents trained in different environments represents a major obstacle for multi-source transfer learning.
To the best of our knowledge, we are the first to propose a solution that allows the combination of models from agents trained in different environments, from which the most relevant information can autonomously be extracted for a given target task.
4.2.1 Universal Feature Space
As the focus of this paper is on world model-based agents, we are required to facilitate a shared feature space across the agents when combining and transferring their components.
Hence, we introduce a universal feature space (UFS) inspired by SEER (Chen et al., 2021), where the authors show that encoders of converged autoencoders can be frozen and reused in visually similar environments due to the generality of convolutional features.
In this paper we make use of two different types of environments: locomotion and pendula tasks (Figure 3).
Therefore, we decide to train a single agent simultaneously in one locomotion and one pendulum environment (as described in Section 4.1.1), such that we learn convolutional features for both types of environments.
After training, we freeze and reuse this agent’s encoder across both source and target agents in order to train them within a UFS.
Note that we do not transfer and freeze the other
RSSM components, as this would prevent the Dreamer agent from learning new reward and transition functions.
As Dreamer learns via reconstruction, the encoder is the main component of the representation model responsible for constructing the feature space.
[Figure] [Figure]
When an agent makes use of the UFS for training in a given target task, we can transfer components from agents that each mastered tasks using the UFS in different environments.
We propose to accomplish this by introducing meta-model transfer learning (MMTL).
For a given component of the target agent, we assemble the same component from all source agents into an ensemble of frozen components, which we connect to the component of the target agent.
In addition to the usual input element(s), this also provides information signals from each of the frozen source components as a means of transfer learning.
Let 𝑚 Θ (𝑦|𝑥) denote a neural network component with param- eters Θ , some input 𝑥 , and some output 𝑦 , belonging to the agent that will be trained on target MDP 𝑀 .
Let 𝑚 𝜃 𝑖
(𝑦|𝑥) de- note the same component belonging to some other agent 𝑖 with frozen parameters 𝜃 𝑖 that were fit to some source MDP 𝑀 𝑖 , where 𝑖 ∈ 𝑁 , and 𝑁 denotes the number source MDPs on which a separate agent was trained on from the set of source MDPs  .
In MMTL, we modify 𝑚 Θ (𝑦|𝑥) , such that we get:
[Equation]
where all models were trained within the same UFS.
Intuitively, information signals provided by the source models can be used to enhance the learning process of 𝑚 Θ .
For instance, assume the objective of 𝑀 𝑖 is similar to the objective in 𝑀 .
In that case, if we use MMTL for the reward model, 𝑚 Θ can autonomously learn to utilize the predictions of 𝑚 𝜃 𝑖 via gradient descent.
Similarly, it can learn to ignore the predictions of source models that provide irrelevant predictions.
As we are dealing with locomotion and pendula environments in this paper, we choose to apply MMTL to the reward model of Dreamer (Figure 2), as the locomotion
MDPs share a similar objective among each other whilst the pendula environments will provide irrelevant signals for the locomotion environments and vice versa.
As such, we can evaluate whether our approach can autonomously learn to utilize and ignore relevant and irrelevant information signals respectively.
Note that in the case of Dreamer’s reward model, 𝑦 corresponds to a scalar Gaussian parameterized by a mean 𝜇 and unit variance, from which the mode is sampled.
Hence, in our case, 𝑦 corresponds to 𝜇 , and 𝑥 corresponds to some latent state 𝑠 .
As such, the target agent will make use of the same frozen encoder used by the source agents, in addition to using a reward meta-model.
We don’t transfer any other components of the architecture in order to be able to observe the isolated effect of the reward meta-model.
Experiments
We evaluate the proposed methods using Dreamer 2 on 6 continuous control tasks: Hopper, Ant, Walker2D,
HalfCheetah, InvertedPendulumSwingup, and InvertedDoublePendulumSwingup (Figure 3).
A detailed description of the differences between the MDPs can be found in Appendix A.
We perform experiments for both multi-task (referred to as FTL) and multiple-agent (referred to as MMTL) transfer learning settings.
For each method, we run multi-source transfer learning experiments using a different set of 𝑁 source tasks for each of the target environments (see Appendix B for the combinations used).
The selection of source tasks for a given set was done such that each source set for a given target environment includes at least one task from a different environment type, i.e. a pendulum task for a locomotion task and vice versa.
Similarly, each source set contains at least one task of the same environment type.
We also ran preliminary experiments (3 random seeds) for sets consisting of 𝑁 =
[2, 3] in order to observe potential performance differences that result from different 𝑁 , but we found no significant differences (see Appendix
C).
To demonstrate that our methods can autonomously extract useful knowledge from a set of source tasks that includes at least one irrelevant source task, we apply our methods to the most challenging setting ( 𝑁 = 4 ) for 9 random seeds.
We used a single Nvidia V100 GPU for each training run, taking about 6 hours per 1 million environment steps.
For each run, we train the FTL and MMTL target agents for 1 million environment steps and compare them to a baseline Dreamer agent that learns from scratch for 1 million environment steps.
FTL is evaluated by training multi-task agents for 2 million environment steps for a single run, after which we transfer the parameters to the target agent as described in Section 4.1.3.
We use a fraction of 𝛼 = 0.2 for FTL, as we observed in preliminary experiments the largest performance gains occur in a range of 𝛼 ∈ [0.1, 0.3] (see
Appendix E).
For creating the UFS for MMTL, we train a multi-task agent on the Hopper and InvertedPendulum task for 2 million environment steps.
We evaluate MMTL by training a single agent on each task of the set of source environments for 1 million environment steps (all using the same UFS), after which we transfer their reward models to the target agent as described in Section 4.2.2.
The overall aggregated return of both FTL and MMTL is reported in Table 1, which allows us to take jumpstarts into account in the results.
The aggregated return of the final 10% (1e5) environment steps can be found in Table 2, allowing us to observe final performance improvements.
Figure 4 shows the corresponding learning curves.
Discussion
We now discuss the results of each of the proposed methods individually and reflect on the overall multisource transfer learning contributions of this paper.
We would like to emphasize that we don’t compare
FTL and MMTL to each other, but to the baseline, as they are used for two entirely different settings.
Additionally, although the proposed techniques are applicable to single-source transfer learning settings as well, empirical performance comparisons between multi-source and single-source transfer learning are outside of the scope of this paper, and we leave this for future work.
Multi-source transfer learning
The transfer learning results show that the proposed multi-source transfer learning methods for both multi-task (FTL) and multiple-agent (MMTL) settings result in positive transfers for 5 out of 6 environments.
We observe jumpstarts, overall performance improvements, and/or final performance gains.
This shows that our proposed methods allow agents to autonomously extract useful information stemming from source agents trained in MDPs with different state-action spaces, reward functions, and transition functions.
We would like to emphasize that for each of the environments, there were at most two environments that could provide useful transfer knowledge.
Nevertheless, our methods are
[Table] [Table]
still able to identify the useful information and result in a positive transfer with 4 source tasks.
For the Ant testing environment
[Table]
we observe negative transfers when directly transferring parameters in the multitask setting, which logically follows from the environment being too different from all other environments in terms of dynamics [Table] as we fully transfer the transition model.
We observe that MMTL does not suffer significantly from this difference in environments, as we don’t transfer the transition model in that case.
Fractional transfer learning Transferring parameters in a modular fashion of world model-based agents that were simultaneously trained on multiple tasks overall results in performance improvements.
For the single pendulum environment, we observe significant jumpstarts (see Figure 4), despite the set of four source tasks merely containing one environment with useful transfer information.
Transferring fractions of parameters plays a major role in these performance improvements as observed in preliminary experiments (Appendix E).
However, one drawback of FTL is that it introduces a tunable fraction parameter 𝛼 , where the optimal fraction can differ per environment and composition of source tasks.
We observed the most performance gains in a range of 𝛼 ∈ [0.1, 0.3] , and for larger fractions the overall transfer learning performance of the agent would degrade.
In terms of deep RL hyperparameters, this can be considered a reasonably short tuning range.
Meta-model transfer learning We observe substantial performance gains for MMTL by merely providing additional information signals to the reward model, next to transferring weights of the encoder model.
To the best of our knowledge, we are the first to successfully combine individual components of multiple individual agents trained in different environments as a multi-source transfer learning technique that results in positive transfers, which is autonomously accomplished through gradient descent.
In addition, the usage of a UFS (which includes a frozen encoder) saves computation.
Conclusion
In this paper, we introduced several techniques that enable the application of multi-source transfer learning to deep MBRL algorithms.
Two major obstacles in applying transfer learning to the RL domain are selecting an optimal source task and the choice regarding which type of transfer learning to apply to individual components.
We addressed these issues by proposing two modular multi-source transfer learning techniques, each applicable to a different cross-domain multi-source situation: a single agent that mastered several tasks, and multiple agents that mastered a single task.
First, we introduced a novel type of transfer learning, which allows fractions of parameters to be transferred, as opposed to discarding information as commonly done by randomly initializing a layer of a neural network.
We used this type of transfer learning as an option in an insightful discussion concerning what type of transfer learning deep MBRL algorithm components benefit from.
The conclusions of this discussion were empirically validated in the multi-task transfer learning setting, which both showed that the modular transfer learning decisions result in significant performance improvements, as well as that we can autonomously extract useful information from multiple different source tasks.
Next, we showed that by learning a universal feature space, we enable the combination and transfer of individual components from agents trained in environments with different state-action spaces, and reward functions.
We extend this concept by introducing meta-model transfer learning, which leverages the predictions of models trained by different agents in addition to the usual input signals, as a multi-source transfer learning technique for multiple-agent settings.
This results in significant sample efficiency gains in challenging cross-domain experiments.
Our aim is to bring RL closer to real-world application and we believe that the techniques introduced in this paper represent an important step toward this direction, by circumventing crucial limitations of transfer learning in RL.
Given the flexibility and modularity of the proposed techniques, we believe they are sufficiently general to potentially be applicable to components of other types of deep RL algorithms.
A Environment Descriptions
In this paper locomotion and pendulum balancing environments from PyBullet (Coumans & Bai, 2016– 2021) are used for experiments.
The locomotion environments have as goal to walk to a target point that is distanced 1 kilometer away from the starting position as quickly as possible.
Each environment has a different entity with different numbers of limbs and therefore has different state-action spaces, and transition functions.
The reward function is similar, slightly adapted for each entity as the agent is penalized for certain limbs colliding with the ground.
The pendula environments have as objective to balance the initially inverted pendulum upwards.
The difference between the two environments used is that one environment has two pendula attached to each other.
This environment is not included in the PyBullet framework for swing-up balancing, which we, therefore, implemented ourselves.
The reward signal for the InvertedPendulumSwingup for a given observation 𝑜 is:
[Equation]
where Θ is the current position of the joint.
For the InvertedDoublePendulumSwingup a swing-up task, we simply add the cosine of the position of the second joint Γ to Equation 1:
[Equation]
As such, these two environments also differ in state-action spaces, transition functions, and reward functions.
B Transfer Learning Task Combinations
In Table 3 the combinations of source tasks and target tasks can be viewed that were used for the experiments in this paper.
That is, they correspond to the results of Appendix C and Section 5.
[Table]
Cheetah [Equation] Hopper [Equation]
Hopper, [Equation] Ant
Cheetah [Equation]
C Preliminary Experiments for 2 and 3 Source Tasks
In Table 4 and Table 5 the results for training on, and transferring, 2 and 3 source tasks as described in
Section 5 can be found for both FTL and MMTL, showing the overall average episode return and the episode return for the final 1e-5 environment steps respectively.
In Figure 5 the corresponding learning curves can be found.
These experiments are the averages and standard deviation across 3 seeds for FTL, MMTL, and a baseline Dreamer trained from scratch.
As just 3 seeds were used for these experiments, they are not conclusive.
However, they do show that regardless of the number of source tasks, our methods can extract useful information and enhance the performance compared to the baseline.
[Table] [Table]
D Full Transfer
In Figure 6 learning curves can be seen for the HalfCheetah task where we fully transfer the parameters instead of using FTL.
As can be seen, this results in a detrimental overfitting effect where the agent does not seem to be learning.
E Additional Fraction Results
In this appendix, results can be found that illustrate the effect of different fractions 𝛼 .
We present the numerical results of transferring fractions of 𝛼 ∈ [0.0, 0.1, 0.2, 0.3, 0.4, 0.5] using 2, 3, and 4 source tasks.
That is, the overall performance over 3 random seeds for the HalfCheetah and Hopper, Walker2D and Ant, and InvertedPendulumSwingup and InvertedDoublePendulumSwingup testing environments can be found in Table 6, Table
[Table]
and Table 8 respectively.
Again, as just 3 seeds were used for these experiments, they are not conclusive.
However, they do indicate that the choice of 𝛼 can largely impact performance, and generally, the best performance gains are yielded with 𝛼 ∈ [0.1, 0.3] .
[Table]
[Table] [Table]