T RANSFER RL ACROSS O BSERVATION F EATURE S PACES VIA M ODEL -B ASED R EGULARIZATION
A BSTRACT
In many reinforcement learning (RL) applications, the observation space is speciﬁed by human developers and restricted by physical realizations, and may thus be subject to dramatic changes over time (e.g. increased number of observable features).
However, when the observation space changes, the previous policy will likely fail due to the mismatch of input features, and another policy must be trained from scratch, which is inefﬁcient in terms of computation and sample complexity.
Following theoretical insights, we propose a novel algorithm which extracts the latent-space dynamics in the source task, and transfers the dynamics model to the target task to use as a model-based regularizer.
Our algorithm works for drastic changes of observation space (e.g. from vector-based observation to image-based observation), without any inter-task mapping or any prior knowledge of the target task.
Empirical results show that our algorithm signiﬁcantly improves the efﬁciency and stability of learning in the target task.
I NTRODUCTION
Deep Reinforcement Learning (DRL) has the potential to be used in many large-scale applications such as robotics, gaming and automotive.
In these real-life scenarios, it is an essential ability for agents to utilize the knowledge learned in past tasks to facilitate learning in unseen tasks, which is known as Transfer RL (TRL).
Most existing TRL works (Taylor & Stone, 2009; Zhu et al., 2020) focus on tasks with the same state-action space but different dynamics/reward.
However, these approaches do not apply to the case where the observation space changes signiﬁcantly.
Observation change is common in practice as in the following scenarios.
(1) Incremental environment development.
RL is used to train non-player characters (NPC) in games (Juliani et al., 2018), which may be frequently updated.
When there are new scenes, characters, or obstacles added to the game, the agent’s observation space will change accordingly.
(2) Hardware upgrade/replacement.
For robots with sensory observations (Bohez et al., 2017), the observation space could change (e.g. from text to audio, from lidar to camera) as the sensor changes.
(3) Restricted data access.
In some
RL applications (Ganesh et al., 2019), agent observation contains sensitive data (e.g. inventory) which may become unavailable in the future due to data restrictions.
In these cases, the learner may have to discard the old policy and train a new policy from scratch, as the policy has a signiﬁcantly different input space, even though the underlying dynamics are similar.
But training an RL policy from scratch can be expensive and unstable.
Therefore, there is a crucial need for a technique that transfers knowledge across tasks with similar dynamics but different observation spaces.
Besides these existing common applications, there are more beneﬁts of across-observation transfer.
For example, observations in real-world environments are usually rich and redundant, so that directly learning a policy is hard and expensive.
If we can transfer knowledge from low-dimensional and informative vector observations (usually available in a simulator) to richer observations, the learning efﬁciency can be signiﬁcantly improved.
Therefore, an effective transfer learning method enables many novel and interesting applications, such as curriculum learning via observation design.
[Figure]
In this paper, we aim to ﬁll the gap and propose a new algorithm that can automatically transfer knowledge from the old environment to facilitate learning in a new environment with a (drastically) different observation space.
In order to meet more practical needs, we focus on the challenging setting where the observation change is: (1) unpredictable (there is no prior knowledge about how the observations change), (2) drastic (the source and target tasks have signiﬁcantly different observation feature spaces, e.g., vector to image), and (3) irretrievable (once the change happens, it is impossible to query the source task, so that the agent can not interact with both environments simultaneously).
Note that different from many prior works (Taylor et al., 2007; Mann & Choe, 2013), we do not assume the knowledge of any inter-task mapping.
That is, the agent does not know which new observation feature is corresponding the which old observation feature.
To remedy the above challenges and achieve knowledge transfer, we make a key observation that, if only the observation features change, the source and target tasks share the same latent space and dynamics (e.g. in Figure 1, O ( S ) and O ( T ) can be associated to the same latent state).
Therefore, we ﬁrst disentangle representation learning from policy learning, and then accelerate the target-task agent by regularizing the representation learning process with the latent dynamics model learned in the source task.
We show by theoretical analysis and empirical evaluation that the target task can be learned more efﬁciently with our proposed transfer learning method than from scratch.
Summary of Contributions.
(1) To the best of our knowledge, we are the ﬁrst to discuss the transfer problem where the source and target tasks have drastically different observation feature spaces, and there is no prior knowledge of an inter-task mapping.
(2) We theoretically characterize what constitutes a “good representation” and analyze the sufﬁcient conditions the representation should satisfy.
(3) Theoretical analysis shows that a model-based regularizer enables efﬁcient representation learning in the target task.
Based on this, we propose a novel algorithm that automatically transfers knowledge across observation representations.
(4) Experiments in 7 environments show that our proposed algorithm signiﬁcantly improves the learning performance of RL agents in the target task.
P RELIMINARIES AND B ACKGROUND
Basic RL Notations.
An RL task can be modeled by a Markov Decision Process (MDP) (Puterman, 2014), deﬁned as a tuple M =
h O , A , P, R,  i , where O is the state/observation space, A is the action space, P is the transition kernel, R is the reward function and  is the discount factor.
At timestep t , the agent observes state o t , takes action a t based on its policy ⇡ : O !
( A ) (where  ( · ) denotes the space of probability distributions), and receives reward r t =
R ( o t , a t ) .
The environment then proceeds to the next state o t +1 ⇠ P ( ·
| o t , a t ) .
The goal of an RL agent is to ﬁnd a policy ⇡ in the policy space ⇧ with the highest cumulative reward, which is characterized by the value functions.
The value of a policy ⇡ for a state o 2 O is deﬁned as V ⇡ ( o ) =
E ⇡ ,P [
P 1 t =
t r t | o
= o ] .
The Q value of a policy ⇡ for a state-action pair ( o, a ) 2 O ⇥
A is deﬁned as Q ⇡ ( o, a ) =
E ⇡ ,P [
P 1 t =
t r t | o
= o, a 0 = a ] .
Appendix A provides more background of RL.
Representation Learning in RL.
Real-world applications usually have large observation spaces for which function approximation is needed to learn the value or the policy.
However, directly learning a policy over the entire observation space could be difﬁcult, as there is usually redundant information in the observation inputs.
A common solution is to map the large-scale observation into a smaller representation space via a non-linear encoder (also called a representation mapping )
: O !
R d , where d is the representation dimension, and then learn the policy/value function over the representation space  ( O ) .
In DRL, the encoder and the policy/value are usually jointly learned.
P ROBLEM S ETUP : T RANSFER A CROSS D IFFERENT O BSERVATION S PACES
We aim to transfer knowledge learned from a source MDP to a target MDP, whose observation spaces are different while dynamics are structurally similar.
Denote the source MDP as M ( S )
=
h O ( S ) , A , P ( S ) , R ( S ) ,  i , and the target MDP as M ( T ) =
h O ( T ) , A , P ( T ) , R ( T ) ,  i .
Note that O ( S ) and
O ( T ) can be signiﬁcantly different, such as O ( S ) being a low-dimensional vector space and O ( T ) being a high-dimensional pixel space, which is challenging for policy transfer since the source target policy have different input shapes and would typically be very different architecturally.
In this work, as motivated in the Introduction, we focus on the setting wherein the dynamics (( P ( S ) , R ( S ) ) and ( P ( T ) , R ( T ) )) of the two MDPs between which we transfer knowledge are deﬁned on different observation spaces but share structural similarities.
Speciﬁcally, we make the assumption that there exists a mapping between the source and target observation spaces such that the transition dynamics under the mapping in the target task share the same transition dynamics as in the source task.
We formalize this in Assumption 1:
Assumption 1.
There exists a function f : O ( T ) !
O ( S ) such that 8 o ( T ) i , o ( T )
j 2 O ( T ) , 8 a 2 A ,
P ( T ) ( o ( T ) j | o ( T ) i , a ) = P ( S ) ( f ( o ( T ) j ) | f ( o ( T ) i ) , a ) ,
R ( T ) ( o ( T ) i , a ) = R ( S ) ( f ( o ( T ) i ) , a ) .
Remarks.
(1) Assumption 1 is mild as many real-world scenarios fall under this assumption.
For instance, when upgrading the cameras of a patrol robot to have higher resolutions, such a mapping f can be a down-sampling function.
Our algorithm does not assume any prior knowledge of the mapping f , although Assumption 1 assumes the existence of such an underlying mapping.
(3) f is a general function without extra restrictions.
f can be a many-to-one mapping, i.e., more than one target observations can be related to the same observation in the source task.
f can be non-surjective, i.e., there could exist source observations that do not correspond to any target observation.
Many prior works (Mann & Choe, 2013; Brys et al., 2015) have similar assumptions, but require prior knowledge of such an inter-task mapping to achieve knowledge transfer.
However, such a mapping might not be available in practice.
As an alternative, in the next section, we propose to learn a latent representation of the observations and a dynamics model in this latent space.
The dynamics model is transferred to speed up learning in the target task, where we do not learn the mapping f explicitly.
M ETHODOLOGY : T RANSFER WITH R EGULARIZED R EPRESENTATION
In this section, we ﬁrst formally characterize “what a good representation is for RL” in Section 4.1, then introduce our proposed transfer algorithm based on representation regularization in Section 4.2, and next provide theoretical analysis of the algorithm in Section 4.3.
C HARACTERIZING C ONDITIONS FOR G OOD R EPRESENTATIONS
As discussed in Section 2, real-world applications usually have rich and redundant observations, where learning a good representation (Jaderberg et al., 2016; Dabney et al., 2020) is essential for efﬁciently ﬁnding an optimal policy.
However, the properties that constitute a good representation for an RL task are still an open question.
Some prior works (Bellemare et al., 2019; Dabney et al., 2020; Gelada et al., 2019) have discussed the representation quality in DRL, but we take a different perspective and focus on characterizing the sufﬁcient properties of representation for learning a task.
Given a representation mapping  , the Q value of any ( o, a ) 2 O ⇥ A can be approximately represented by a function of  ( o ) , i.e., ˆ Q ( o, a ) =
f (  ( o ); ✓ a ) , where ✓ a is a set of learnable parameters.
For example, if linear approximation is used, the Q value can be represented as ˆ Q ( o, a ) =
( o ) ✓ a , where ✓ a 2 R d is a vector associated with action a .
To study the relation between representation quality and approximation quality, we deﬁne an approximation operator H  , which approximates the Q value with hypothesis class H .
Formally, let ⇥ denote the parameter space of functions in H , then H  Q ( o ) :
=
h (  ( o ); ✓ ⇤ ) , where ✓ ⇤ = argmin ✓ 2 ⇥ E [ k h (  ( o ); ✓ )  Q (  , · ) k ] .
Such an approximation can be realized by neural networks as universal function approximators (Hornik et al., 1989).
Therefore, the approximation error k Q  ˆ Q k only depends on the selection of the representation  .
The quality of the encoder  is crucial for learning an accurate value function or learning a good policy.
The ideal encoder  should discard irrelevant information in the raw observation but keep essential information.
In supervised or self-supervised representation learning (Chen et al., 2020;
Achille & Soatto, 2018), it is believed that a good representation  ( X ) of input X should contain minimal information of X which maintaining sufﬁcient information for predicting the label Y .
However, in RL, it is difﬁcult to identify whether a representation is sufﬁcient, since there is no label corresponding to each input.
The focus of an agent is to estimate the value of each input o 2 O , which is associated with some policy.
Therefore, we point out that the representation quality in RL is policy-dependent .
Below, we formally characterize the sufﬁciency of a representation mapping in terms of a ﬁxed policy and learning a task.
Sufﬁciency for A Fixed Policy.
If the agent is executing a ﬁxed policy, and its goal is to estimate the expected future return from the environment, then a representation is sufﬁcient for the policy as long as it can encode the policy value V ⇡ .
A formal deﬁnition is provided by Deﬁnition 9 in Appendix B.
Sufﬁciency for Learning A Task.
The goal of RL is to ﬁnd an optimal policy.
Therefore, it is not adequate for the representation to only ﬁt one policy.
Intuitively, a representation mapping is sufﬁcient for learning if we are able to ﬁnd an optimal policy over the representation space  ( O ) , which requires multiple iterations of policy evaluation and policy improvement.
Deﬁnition 2 below deﬁnes a set of “important” policies for learning with  ( O ) .
Deﬁnition 2 (Encoded Deterministic Policies) .
For a given representation mapping  ( · ) , deﬁne an encoded deterministic policy set ⇧ D  as the set of policies that are deterministic and take the same actions for observations with the same representations.
Formally, ⇧ D  := { ⇡ 2 ⇧ : 9 ˜ ⇡ :  ( O ) !
A , s.t.
o 2 O , ⇡ ( o ) =
˜ ⇡
(  ( o )) } .
A policy ⇡ is in ⇧ D  if it does not distinguish o 1 and o 2 when  ( o 1 )
=
( o 2 ) .
Therefore, ⇧ D  can be regarded as deterministic policies that make decisions for encoded observations.
Now, we deﬁne the concept of sufﬁcient representation for learning in an MDP.
Deﬁnition 3 (Sufﬁcient Representation for Learning) .
A representation mapping  is sufﬁcient for a task M w.r.t.
approximation operator H
if H  Q ⇡ = Q ⇡ , for all ⇡ 2 ⇧ D  .
(  is ✏ -sufﬁcient for learning if k H  Q ⇡  Q ⇡ k  ✏ and linearly-sufﬁcient if H  is linear transformation.)
Deﬁnition 3 suggests that the representation is sufﬁcient for learning a task as long as it is sufﬁcient for policies in ⇧ D  .
Then, the lemma below justiﬁes that a nearly sufﬁcient representation can ensure that approximate policy iteration converges to a near-optimal solution.
Lemma 4 (Error Bound for Approximate Policy Iteration) .
If  is ✏ -sufﬁcient for task M (with `norm), then the approximated policy iteration with approximation operator H  starting from any initial policy that is encoded by  ( ⇡ 0 2 ⇧ D  ) satisﬁes
[Equation]
where ⇡ k is the policy in the k -th iteration.
Lemma 4, proved in Appendix C, is extended from the error bound provided by Bertsekas & Tsitsiklis (1996).
For simplicity, we consider the bound in ` 1 , but tighter bounds can be derived with other norms (Munos, 2005), although a tighter bound is not the focus of this paper.
How Can We Learn A Sufﬁcient Representation?
So far we have provided a principle to deﬁne whether a given representation is sufﬁcient for learning.
In DRL, the representation is learned together with the policy or value function using neural networks, but the quality of the representation may be poor (Dabney et al., 2020), which makes it hard for the agent to ﬁnd an optimal policy.
Based on Deﬁnition 3, a natural method to learn a good representation is to let the representation ﬁt as many policy values as possible as auxiliary tasks, which matches the ideas in other works.
For example, Bellemare et al.
(2019) propose to ﬁt a set of representative policies (called adversarial value functions ).
Dabney et al.
(2020) choose to ﬁt the values of all past policies (along the value
[Table]
L P and L R deﬁned
Equation (2)
[Table]
[Table]
[Table]
[Table]
[Table]
improvement path ), which requires less computational resource.
Different from these works that directly ﬁt the value functions of multiple policies, in Section 4.2, we propose a computationally efﬁcient way to realize sufﬁcient representation for learning and knowledge transferring: to ﬁt and transfer an auxiliary policy-independent dynamics model.
4. A LGORITHM : L EARNING AND T RANSFERRING M ODEL - BASED R EGULARIZER
Our goal is to use the knowledge learned in the source task to learn a good representation in the target task, such that the agent learns the target task more easily than learning from scratch.
Since we focus on developing a generic transfer mechanism, the base learner can be any DRL algorithms.
We use L base to denote the loss function of the base learner.
As motivated in Section 4.1, we propose to learn policy-independent dynamics models for producing high-quality representations: (1) ˆ P which predicts the representation of the next state for a given state action pair, and (2) ˆ R which predicts the immediate reward for a given state-action pair.
For a batch of N transition samples { o i , a i , o 0 i , r i } Ni =1 , deﬁne the transition loss and the reward loss as:
[Equation]
where ¯  ( o 0 i ) denotes the representation of the next state o
i with stop gradients.
In order to ﬁt a more diverse state distribution, transition samples are drawn from an off-policy buffer, which stores shufﬂed past trajectories.
Note that a similar model loss is commonly used in model-based RL (van der
Pol et al., 2020) and bisimulation works Zhang et al.
(2020b),
but our focus is to use the learned model as an auxiliary task and to transfer the model to the target task. Encoder Policy/Value Dynamics
[Figure]
The learning procedures for the source task and the target task are illustrated in Algorithm 1 and Algorithm 2, respectively.
Figure 2 depicts the architecture of the learning model for both source and target tasks.
z =  ( o ) and z 0 =
¯  ( o 0 ) are the encoded observation and next observation.
Given the current encoding z and the action a , the dynamics models ˆ P and ˆ R return the predicted next encoding ˆ z
= ˆ P ( z, a ) and predicted reward ˆ r = ˆ R ( z, a ) .
Then the transition loss is the mean squared error (MSE) between z 0 and ˆ z 0 in a batch; the reward loss is the MSE between r and ˆ r in a batch.
In the source task (Algorithm 1): dynamics models ˆ P and ˆ R are learned by minimizing L P and
L R , which are computed based on a recent copy of encoder called stable encoder ˆ  ( S ) (Line 5).
The computation of the stable encoder is to help the dynamics models converge, as the actual encoder  ( S ) changes at every step.
Note that a stable copy of the network is widely used in many DRL algorithms (e.g. the target network in DQN), which can be directly regarded as ˆ  ( S ) without maintaining an extra network.
The actual encoder  ( S ) is regularized by the auxiliary dynamics models ˆ P and ˆ R (Line 6).
In the target task (Algorithm 2): dynamics model ˆ P and ˆ R are transferred from the source task and ﬁxed during learning.
Therefore, the learning of  ( T ) is regularized by static dynamics models, which leads to faster and more stable convergence than naively learning an auxiliary task.
Relation and Difference with Model-based RL and Bisimulation Metrics.
The goal of modelbased RL (Kipf et al., 2019; Grimm et al., 2020) is to learn an accurate world model and use the model for planning.
The dynamics model could be learned on either raw observations or representations.
In our framework, we also learn a dynamics model, but the model serves as an auxiliary task, and learning is still performed by the model-free base learner with L base .
Bisimulation metrics (Castro, 2020; Zhang et al., 2020b) aim to approximate the distances among states by learning dynamics models, whereas we do not explicitly measure the distance among states.
Note that we also do not require a reconstruction loss that is common in literature (Lee et al., 2019).
4. T HEORETICAL A NALYSIS : B ENEFITS OF T RANSFERABLE D YNAMICS M ODEL
The algorithms introduced in Section 4.2 consist of two designs: learning a latent dynamics model as an auxiliary task, and transferring the dynamics model to the target task.
In this section, we show theoretical justiﬁcations and practical advantages of our proposed method.
We aim to answer the following two questions: (1) How does learning an auxiliary dynamics model help with representation learning?
(2) Is the auxiliary dynamics model transferable?
For notational simplicity, let P a and R a denote the transition and reward functions associated with action a 2 A .
Note that P a
and R a are independent of any policy.
We then deﬁne the sufﬁciency of a representation mapping w.r.t.
dynamics models as below.
Deﬁnition 5 (Policy-independent Model Sufﬁciency) .
For an MDP M , a representation mapping  is sufﬁcient for its dynamics ( P a , R a ) a 2 A if 8 a 2 A , there exists functions ˆ P a : R d !
R d and ˆ
R a : R d !
R such that 8 o 2 O , ˆ P a (  ( o )) =
E o
⇠ P a ( o )
[  ( o 0 )] , ˆ R a (  ( o ))
=
R a ( o ) .
Remarks.
(1) Since we focus on learning a deterministic transition prediction model ˆ P ,  is exactly sufﬁcient for dynamics ( P a , R a ) a 2 A only when the transition function P is deterministic.
If
P is stochastic, but we have max o,a k E o
⇠ P a ( o )
[  ( o 0 )]
ˆ P a (  ( o ))
k  ✏ P and max o,a | R a ( o )  ˆ
R a (  ( o )) |  ✏ R , then  is ( ✏ P , ✏ R ) -sufﬁcient for the dynamics of M .
Next we show by Proposition 6 and Theorem 7 that learning sufﬁciency can be achieved via ensuring model sufﬁciency.
Proposition 6 (Learning Sufﬁciency Induced by Policy-independent Model Sufﬁciency) .
Consider an MDP M with deterministic transition function P and reward function R .
If  is sufﬁcient for ( P a , R a ) a 2 A , then it is sufﬁcient (but not necessarily linearly sufﬁcient) for learning in M .
Proposition 6 shows that, if the transition is deterministic and the model errors L P , L R are zero, then  is exactly sufﬁcient for learning.
More generally, if the transition function P is not deterministic, and model ﬁtting is not perfect, the learned representation can still be nearly sufﬁcient for learning as characterized by Theorem 7 below, which is extended from a variant of the value difference bound derived by Gelada et al.
Proposition 6 and Theorem 7 justify that learning the latent dynamics model as an auxiliary task encourages the representation to be sufﬁcient for learning.
The model error L P and L R deﬁned in Section 4.2 can indicate how good the representation is.
Theorem 7.
For an MDP M , if representation mapping  is ( ✏ P , ✏ R ) -sufﬁcient for the dynamics of M , then approximate policy iteration with approximation operator H  starting from any initial policy ⇡ 0 2 ⇧ D  satisﬁes
[Equation]
where K  ,V is an upper bound of the value Lipschitz constant as deﬁned in Appendix B.
Transferring Model to Get Better Representation in Target.
Although Proposition 6 shows that learning auxiliary dynamics models beneﬁts representation learning, ﬁnding the optimal solution is non-trivial since one still has to learn ˆ P and ˆ R .
Therefore, the main idea of our algorithm is to transfer the dynamics models ˆ P, ˆ R from the source task to the target task, to signiﬁcantly ease the learning in the target task.
Theorem 8 below guarantees that transferring the dynamics models is feasible.
Our experimental result in Section 6 veriﬁes that learning with transferred and ﬁxed dynamics models outperforms learning with randomly initialized dynamics models.
Theorem 8 (Transferable Dynamics Models) .
Consider a source task M ( S ) and a target task M ( T ) with deterministic transition functions.
Suppose  ( S ) is sufﬁcient for ( P ( S ) a , R ( S ) a ) a 2 A with functions
ˆ P a , ˆ R a , then there exists a representation  ( T ) satisfying ˆ P a (  ( o ))
=
E o 0 ⇠ P ( T ) a [  ( o 0 )] , ˆ R a (  ( o )) =
R ( T ) a ( o ) , for all o 2 O ( T ) , and  ( T ) is sufﬁcient for learning in M ( T ) .
Theorem 8 shows that the learned latent dynamics models ˆ P, ˆ R are transferable from the source task to the target task.
For simplicity, Theorem 8 focuses on exact sufﬁciency as in Proposition 6, but it can be easily extended to ✏ -sufﬁciency if combined with Theorem 7.
Proofs for Proposition 6,
Theorem 7 and Theorem 8 are all provided in Appendix C.
Trade-off between Approximation Complexity and Representation Complexity.
As suggested by Proposition 6, ﬁtting policy-independent dynamics encourages the representation to be sufﬁcient for learning, but not necessarily linearly sufﬁcient.
Therefore, we suggest using a non-linear policy/value head following the representation to reduce the approximation error.
Linear sufﬁciency can be achieved if  is made linearly sufﬁcient for P ⇡ and R ⇡ for all ⇡ 2 ⇧ D  , where P ⇡ and R ⇡ are transition and reward functions induced by policy ⇡ (Proposition 10, Appendix B).
However, using this method for transfer learning is expensive in terms of both computation and memory, as it requires to learn P ⇡ and R ⇡ for many different ⇡ ’s and store these models for transferring to the target task.
Therefore, there is a trade-off between approximation complexity and representation complexity.
Learning a linearly sufﬁcient representation reduces the complexity of the approximation operator.
But it requires more complexity in the representation itself as it has to satisfy much more constraints.
To develop a practical and efﬁcient transfer method, we use a slightly more complex approximation operator (non-linear policy head) while keeping the auxiliary task simple and easy to transfer across tasks.
Please see Appendix B for more detailed discussion about linear sufﬁciency.
R ELATED W ORK
Transfer RL across Observation Feature Spaces.
Transferring knowledge between tasks with different observation spaces has been studied for years.
Many existing approaches(Taylor et al., 2007; Mann & Choe, 2013;
Brys et al., 2015) require an explicit mapping between the source and target observation spaces, which may be hard to obtain in practice.
Raiman et al.
(2019) introduce network surgery that deals with the change in the input features by determining which components of a neural network model should be transferred and which require retraining.
However, it requires knowledge of the input feature maps, and is not designed for drastic changes, e.g. vector to pixel.
Sun et al.
(2020) propose a provably sample-efﬁcient transfer learning algorithm that works for different observation spaces without knowing any inter-task mapping, but the algorithm is mainly designed for tabular RL and model-based RL which uses the model to plan for a policy, different from our setting.
Gupta et al.
(2017) achieve transfer learning between two different tasks by learning an invariant feature space, with a key time-based alignment assumption.
We empirically compared this method with our proposed transfer algorithm in Section 6.
Our work is also related to state abstraction in block MDPs, as studied by Zhang et al.
(2020a).
But the problem studied in Zhang et al.
(2020a) is a multi-task setting where the agent aims to learn generalizable abstract states from a series of tasks.
Another related topic is domain adaptation in RL (Higgins et al., 2017;
Eysenbach et al., 2020; Zhang et al., 2020b), where the target observation space (e.g. real world) is different from the source observation (e.g. simulator).
However, domain adaptation does not assume drastic observation changes (e.g. changed dimension).
Moreover, the aim of domain adaptation is usually zero-shot generalization to new observations, thus prior knowledge or a few samples of the target domain is often needed (Eysenbach et al., 2020).
Representation Learning in RL.
In environments with rich observations, representation learning is crucial for the efﬁciency of RL methods.
Learning unsupervised auxiliary tasks (Jaderberg et al., 2016) is shown to be effective for learning a good representation.
The relationship between learning policy-dependent auxiliary tasks and learning good representations has been studied in some prior works (Bellemare et al., 2019; Dabney et al., 2020; Lyle et al., 2021), while our focus is to learn policy-independent auxiliary tasks to facilitate transfer learning.
Using latent prediction models to regularize representation has been shown to be effective for various types of rich observations (Guo et al., 2020; Lee et al., 2019).
Gelada et al.
(2019) theoretically justify that learning latent dynamics model guarantees the quality of the learned representation, while we further characterize the relationship between representation and learning performance, and we utilize dynamics models to improve transfer learning.
Zhang et al.
(2020b) use a bisimulation metric to learn latent representations that are invariant to task-irrelevant details in observation.
As pointed out by Achille & Soatto (2018), invariant and sufﬁcient representation is indeed minimal sufﬁcient, so it is an interesting future direction to combine our method with bisimulation metric to learn minimal sufﬁcient representations.
There is also a line of work using contrastive learning to train an encoder for pixel observations (Srinivas et al., 2020; Yarats et al., 2021; Stooke et al., 2021), which usually pre-train an encoder based on image samples using self-supervised learning.
However, environment dynamics are usually not considered during pre-training.
Our algorithm can be combined with these contrastive learning approaches to further improve learning performance in the target task.
E XPERIMENTAL E VALUATION
We empirically evaluate our transfer learning algorithm in various environments and multiple observation-change scenarios.
Detailed experiment setup and hyperparameters are in Appendix E.
Baselines.
To verify the effectiveness of our proposed transfer learning method, we compare our transfer learning algorithm with 4 baselines:
(1) Single : a single-task base learner.
(2) Auxiliary : learns auxiliary models from scratch to regularize representation.
(3) Fine-tune : loads and freezes the source policy head, and retrains an encoder in the target task.
(4) Time-aligned (Gupta et al., 2017): supposes the target task and the source task proceed to the same latent state given the same action sequence, and pre-trains a target-task encoder with saved source-task trajectories.
More details of baseline implementations are in Appendix E.1.1.
Scenarios.
As motivated in Section 1, there are many scenarios where one can beneﬁt from transfer learning across observation feature spaces.
We evaluate our proposed transfer algorithm in 7 environments that ﬁt various scenarios, to simulate real-world applications: (1) Vec-to-pixel: a novel and challenging scenario, where the source task has low-dimensional vector observations and the target task has pixel observations.
We use 3 vector-input environments CartPole, Acrobot and Cheetah-Run as source tasks, and use the rendered image in the target task.
(2) More-sensor: another challenging scenario where the target task has a lot more sensors than the source task.
We use 3 MuJoCo environments: HalfCheetah, Hopper and Walker2d, whose original observation dimensions are 17, 11 and 17, respectively.
We add mass-based inertia and velocity (provided by MuJoCo’s API), resulting in 145, 91, 145 dimensions in the corresponding target tasks.
(3) Broken-sensor: we use an existing game 3DBall contained in the Unity ML-Agents Toolkit (Juliani et al., 2018), which has two different observation speciﬁcations that naturally ﬁt our transfer setting: the source observation has 8 features containing the velocity of the ball; the target observation does not have the ball’s velocity, thus the agent has to stack the past 9 frames to infer the velocity.
Please see Appendix E.1.2 for more detailed descriptions of all the 7 environments.
Base DRL Learners.
What we propose is a transfer learning mechanism that can be combined with any existing DRL methods.
For environments with discrete action spaces (CartPole, Acrobot), we use the DQN algorithm (Mnih et al., 2015), while for environments with continuous action spaces (Cheetah-Run, HalfCheetah, Hopper, Walker2d, 3DBall), we use the SAC algorithm (Haarnoja et al., 2018).
To ensure a fair comparison, we use the same base DRL learner with the same hyperparameter settings for all tested methods, as detailed in Appendix E.1.3.
As is common in prior works, our implementation of the RL algorithms is mostly a proof of concept, thus many advanced training techniques are not included (e.g. Rainbow DQN).
Results.
Experimental results on all tested environments are shown in Figure 3.
We can see that our proposed transfer method learns signiﬁcantly better than the single-task learner, and also outperforms all baselines in the challenging target tasks.
Our transfer method outperforms Auxiliary since it transfers dynamics model from the source task instead of learning it from scratch, and outperforms Fine-tine since it regularizes the challenging encoder learning with a model-based regularizer.
[Figure]
The Time-aligned method, although requires additional pre-training that is not shown in the ﬁgures, does not work better than Single in most environments, because the time-based alignment assumption may not hold as discussed in Appendix E.1.1.
In some environments (e.g. Hopper, Walker2d, 3DBall), our transfer algorithm even achieves better asymptotic performance than the source-task policy, which suggests that our method can be used for improving the policy with incremental observation design.
To the best of our knowledge, we are the ﬁrst to achieve effective knowledge transfer from a vector-input environment to a pixel-input environment without any pre-deﬁned mappings.
[Figure]
Ablation Study and Hyper-parameter Test.
To verify the effectiveness of proposed transfer method, we conduct ablation study and compare our method with its two variants: only transferring the transition model ˆ P and only transferring the reward model ˆ R .
Figure 4 shows the comparison in HalfCheetah, and Appendix E.2 demonstrates more results.
We ﬁnd that all the variants of our method can make some improvements, which suggests that transferring ˆ P and ˆ R are both effective designs for accelerating the target task learning.
Figurein Appendix E.2 shows another ablation study where we investigate different selections of model regularizers and policy heads.
In Algorithm 2, a hyper-parameter  is needed to control the weight of the transferred model-based regularizer.
Figure 7 in Appendix E.2 shows that, for a wide range of  ’s, the agent consistently outperforms the single-task learner.
Potential Limitations and Solutions.
As Figure 3 shows, in some environments such as HalfCheetah, our transfer algorithm signiﬁcantly outperforms baselines without transfer.
But in Walker2d, the improvement is less signiﬁcant, although transferring is still better than not transferring.
This phenomenon is common in model-based learning (Nagabandi et al., 2018), as state predicting in
Walker2d is harder than that in HalfCheetah due to the complexity of the dynamics.
Therefore, we suggest using our method to transfer when the learned models ( ˆ P, ˆ R ) in the source task are relatively good (error is low).
More techniques of improving model-based learning, such as bisimulation (Zhang et al., 2020b; Castro, 2020), can be applied to further improve the transfer performance.
C ONCLUSION
In this paper, we identify and propose a solution to an important but rarely studied problem: transferring knowledge between tasks with drastically different observation spaces where inter-task mappings are not available.
We propose to learn a latent dynamics model in the source task and transfer the model to the target task to facilitate representation learning.
Theoretical analysis and empirical study justify the effectiveness of the proposed algorithm.
A CKNOWLEDGEMENTS
This work is supported by Unity Technologies, National Science Foundation IIS-1850220 CRII
Award 030742-00001, DOD-DARPA-Defense Advanced Research Projects Agency Guaranteeing
AI Robustness against Deception (GARD), and Adobe, Capital One and JP Morgan faculty fellowships.