T RANSFER RL ACROSS O BSERVATION F EATURE S PACES VIA M ODEL -B ASED R EGULARIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
In many reinforcement learning (RL) applications, the observation space is speciﬁed by human developers and restricted by physical realizations, and may thus be subject to dramatic changes over time (e.g. increased number of observable features).
However, when the observation space changes, the previous policy will likely fail due to the mismatch of input features, and another policy must be trained from scratch, which is inefﬁcient in terms of computation and sample complexity.
Following theoretical insights, we propose a novel algorithm which extracts the latent-space dynamics in the source task, and transfers the dynamics model to the target task to use as a model-based regularizer.
Our algorithm works for drastic changes of observation space (e.g. from vector-based observation to image-based observation), without any inter-task mapping or any prior knowledge of the target task.
Empirical results show that our algorithm signiﬁcantly improves the efﬁciency and stability of learning in the target task.
I NTRODUCTION
Deep Reinforcement Learning (DRL) has the potential to be used in many large-scale applications such as robotics, gaming and automotive.
In these real-life scenarios, it is an essential ability for agents to utilize the knowledge learned in past tasks to facilitate learning in unseen tasks, which is known as Transfer RL (TRL).
Most existing TRL works (Taylor & Stone, 2009; Zhu et al., 2020) focus on tasks with the same state-action space but different dynamics/reward.
However, these approaches do not apply to the case where the observation space changes signiﬁcantly. Observation change is common in practice as in the following scenarios. (1) Incremental environment development.
RL is used to train Non-player characters (NPC) in games (Juliani et al., 2018), which may be frequently updated. When there are new scenes, characters, or obstacles added to the game, the agent’s observation space will change accordingly. (2) Hardware upgrade/replacement.
For robots with sensory observations (Bohez et al., 2017), the observation space could change (e.g. from text to audio, from lidar to camera) as the sensor changes. (3) Restricted data access. In some RL applications (Ganesh et al., 2019), agent observation contains sensitive data (e.g. inventory) which may become unavailable in the future due to data restrictions.
In these cases, the learner may have to discard the old policy and train a new policy from scratch, as the policy has a signiﬁcantly different input space, even though the underlying dynamics are similar. But training an RL policy from scratch can be expensive and unstable.
Therefore, there is a crucial need for a technique that transfers knowledge across tasks with similar dynamics but different observation spaces.
Besides these existing common applications, there are more beneﬁts of across-observation transfer.
For example, observations in real-world environments are usually rich and redundant, so that directly learning a policy is hard and expensive.
If we can transfer knowledge from low-dimensional and informative vector observations (usually available in a simulator) to richer observations, the learning efﬁciency can be signiﬁcantly improved.
Therefore, an effective transfer learning method enables many novel and interesting applications, such as curriculum learning via observation design.
In this paper, we aim to ﬁll the gap and propose a new algorithm that can automatically transfer knowledge from the old environment to facilitate learning in a new environment with a (drastically) different observation space.
In order to meet more practical needs, we focus on the challenging setting where the observation change is: (1) unpredictable (there is no prior knowledge about how the observations change), (2) drastic (the source and target tasks have signiﬁcantly different
[Figure]
observation feature spaces, e.g., vector to image), and (3) irretrievable (once the change happens, it is impossible to query the source task, so that the agent can not interact with both environments simultaneously). Note that different from many prior works (Taylor et al., 2007; Mann & Choe, 2013), we do not assume the knowledge of any inter-task mapping. That is, the agent does not know which new observation feature is corresponding the which old observation feature.
To remedy the above challenges and achieve knowledge transfer, we make a key observation that, if only the observation features change, the source and target tasks share the same latent space and dynamics (e.g. in Figure 1, O ( S ) and O ( T ) can be associated to the same latent state).
Therefore, we ﬁrst disentangle representation learning from policy learning, and then accelerate the target-task agent by regularizing the representation learning process with the latent dynamics model learned in the source task.
We show by theoretical analysis and empirical evaluation that the target task can be learned more efﬁciently with our proposed transfer learning method than from scratch.
Summary of Contributions.
(1) To the best of our knowledge, we are the ﬁrst to discuss the transfer problem where the source and target tasks have drastically different observation feature spaces, and there is no prior knowledge of an inter-task mapping.
(2) We theoretically characterize what constitutes a “good representation” and analyze the sufﬁcient conditions the representation should satisfy.
(3) Theoretical analysis shows that a model-based regularizer enables efﬁcient representation learning in the target task.
Based on this, we propose a novel algorithm that automatically transfers knowledge across observation representations.
(4) Experiments in 7 environments show that our proposed algorithm signiﬁcantly improves the learning performance of RL agents in the target task.
P RELIMINARIES AND B ACKGROUND
Basic RL Notations. An RL task can be modeled by a Markov Decision Process (MDP) (Puterman, 2014), deﬁned as a tuple M = (cid:104)O , A , P, R, γ (cid:105) , where O is the state/observation space, A is the action space, P is the transition kernel, R is the reward function and γ is the discount factor.
At timestep t , the agent observes state o t , takes action a t based on its policy π : O → ∆( A ) (where ∆( · ) denotes the space of probability distributions), and receives reward r t =
R ( o t , a t ) .
The environment then proceeds to the next state o t +1 ∼ P ( ·
| o t , a t ) .
The goal of an RL agent is to ﬁnd a policy π in the policy space Π with the highest cumulative reward, which is characterized by the value functions.
The value of a policy π for a state o ∈ O is deﬁned as V π ( o ) =
E π,P [ (cid:80)
∞
t =0 γ t r t | o
= o ] .
The Q value of a policy π for a state-action pair ( o, a ) ∈ O ×
A is deﬁned as Q π ( o, a ) =
E π,P [ (cid:80)
∞
t =0 γ t r t | o
= o, a 0 = a ] .
Appendix A provides more background of RL.
Representation Learning in RL.
Real-world applications usually have large observation spaces for which function approximation is needed to learn the value or the policy.
However, directly learning a policy over the entire observation space could be difﬁcult, as there is usually redundant information in the observation inputs.
A common solution is to map the large-scale observation into a smaller representation space via a non-linear encoder (also called a representation mapping ) φ : O → R d , where d is the representation dimension, and then learn the policy/value function over the representation space φ ( O ) .
In DRL, the encoder and the policy/value are usually jointly learned.
P ROBLEM S ETUP : T RANSFER A CROSS D IFFERENT O BSERVATION S PACES
We aim to transfer knowledge learned from a source MDP to a target MDP, whose observation spaces are different while dynamics are structurally similar.
Denote the source MDP as M ( S ) =
(cid:104)O ( S ) , A , P ( S ) , R ( S ) , γ (cid:105) , and the target MDP as M ( T ) =
(cid:104)O ( T ) , A , P ( T ) , R ( T ) , γ (cid:105) .
Note that O ( S ) and
O ( T ) can be signiﬁcantly different, such as O ( S ) being a low-dimensional vector space and O ( T ) being a high-dimensional pixel space, which is challenging for policy transfer since the source target policy have different input shapes and would typically be very different architecturally.
In this work, as motivated in the Introduction, we focus on the setting wherein the dynamics (( P ( S ) , R ( S ) ) and ( P ( T ) , R ( T ) )) of the two MDPs between which we transfer knowledge are deﬁned on different observation spaces but share structural similarities.
Speciﬁcally, we make the assumption that there exists a mapping between the source and target observation spaces such that the transition dynamics under the mapping in the target task share the same transition dynamics as in the source task.
We formalize this in Assumption 1:
Assumption 1.
There exists a function f : O ( T ) → O ( S ) such that ∀ o ( T ) i , o ( T )
j ∈ O ( T ) , ∀ a ∈ A ,
[Equation]
Remarks.
(1) Assumption 1 is mild as many real-world scenarios fall under this assumption.
For instance, when upgrading the cameras of a patrol robot to have higher resolutions, such a mapping f can be a down-sampling function.
Our algorithm does not assume any prior knowledge of the mapping f , although Assumption 1 assumes the existence of such an underlying mapping.
(3) f is a general function without extra restrictions.
f can be a many-to-one mapping, i.e., more than one target observations can be related to the same observation in the source task.
f can be non-surjective, i.e., there could exist source observations that do not correspond to any target observation.
Many prior works (Mann & Choe, 2013; Brys et al., 2015) have similar assumptions, but require prior knowledge of such an inter-task mapping to achieve knowledge transfer.
However, such a mapping might not be available in practice.
As an alternative, in the next section, we propose to learn a latent representation of the observations and a dynamics model in this latent space.
The dynamics model is transferred to speed up learning in the target task, where we do not learn the mapping f explicitly.
M ETHODOLOGY : T RANSFER WITH R EGULARIZED R EPRESENTATION
The main idea of our proposed transfer method is to learn a latent dynamics model in the source task, and use the dynamics model to regularize representation learning in the target task. This section is organized as follows.
First, we formally characterize “what a good representation is for RL” in Section 4.1, which is crucial to motivate our algorithm.
Then, we introduce our proposed algorithm for across-observation transfer in Section 4.2, where we learn a latent dynamics model as an auxiliary task in the source task, and transfer the dynamics model to the target task to regularize representation learning. Theoretical analysis for the proposed algorithm is provided in Section 4.3, which explains why the proposed method works and provides insights for the architecture design.
4. C HARACTERIZING C ONDITIONS FOR G OOD R EPRESENTATIONS
As discussed in Section 2, real-world applications usually have rich and redundant observations, where learning a good representation (Jaderberg et al., 2016; Dabney et al., 2020) is essential for efﬁciently ﬁnding an optimal policy.
However, the properties that constitute a good representation for an RL task are still an open question.
Some prior works (Bellemare et al., 2019; Dabney et al., 2020; Gelada et al., 2019) have discussed the representation quality in DRL, but we take a different perspective and focus on characterizing the sufﬁcient properties of representation for learning a task.
Given a representation mapping φ , the Q value of any ( o, a ) ∈
O ×
A can be approximately represented by a function of φ ( o ) , i.e., ˆ Q ( o, a ) =
f ( φ ( o ); θ a ) , where θ a is a set of learnable parameters.
For example, if linear approximation is used, the Q value can be represented as ˆ Q ( o, a ) =
φ ( o ) θ a , where θ a ∈ R d is a vector associated with action a .
To study the relation between representation quality and approximation quality, we deﬁne an approximation operator H φ , which approximates the Q value with hypothesis class H .
Formally, let Θ denote the parameter space of functions in H , then H φ Q ( o ) :
= h ( φ ( o ); θ ∗ ) , where θ ∗ = argmin θ ∈ Θ E [ (cid:107) h ( φ ( o ); θ ) − Q ( φ, · ) (cid:107) ] .
Such an approximation can be realized by neural networks as universal function approximators (Hornik et al., 1989).
Therefore, the approximation error (cid:107) Q − ˆ Q (cid:107) only depends on the selection of the representation φ .
The quality of the encoder φ is crucial for learning an accurate value function or learning a good policy.
The ideal encoder φ should discard irrelevant information in the raw observation but keep essential information.
In supervised or self-supervised representation learning (Chen et al., 2020;
Achille & Soatto, 2018), it is believed that a good representation φ ( X ) of input X should contain minimal information of X which maintaining sufﬁcient information for predicting the label Y .
However, in RL, it is difﬁcult to identify whether a representation is sufﬁcient, since there is no label corresponding to each input.
The focus of an agent is to estimate the value of each input o ∈ O , which is associated with some policy.
Therefore, we point out that the representation quality in RL is policy-dependent .
Below, we formally characterize the sufﬁciency of a representation mapping in terms of a ﬁxed policy and learning a task.
Sufﬁciency for A Fixed Policy. If the agent is executing a ﬁxed policy, and its goal is to estimate the expected future return from the environment, then a representation is sufﬁcient for the policy as long as it can encode the policy value V π .
A formal deﬁnition is provided by Deﬁnition 9 in Appendix B.
Sufﬁciency for Learning A Task. The goal of RL is to ﬁnd an optimal policy.
Therefore, it is not adequate for the representation to only ﬁt one policy.
Intuitively, a representation mapping is sufﬁcient for learning if we are able to ﬁnd an optimal policy over the representation space φ ( O ) , which requires multiple iterations of policy evaluation and policy improvement.
Deﬁnition 2 below deﬁnes a set of “important” policies for learning with φ ( O ) .
Deﬁnition 2 (Encoded Deterministic Policies) .
For a given representation mapping φ ( · ) , deﬁne an encoded deterministic policy set Π Dφ as the set of policies that are deterministic and take the same actions for observations with the same representations.
Formally, Π Dφ := { π ∈ Π : ∃ ˜ π : φ ( O ) → A , s.t. ∀ o ∈ O , π ( o )
= ˜ π ( φ ( o )) } .
A policy π is in Π
Dφ if it does not distinguish o 1 and o 2 when φ ( o 1 ) =
φ ( o 2 ) .
Therefore, Π Dφ can be regarded as deterministic policies that make decisions for encoded observations.
Now, we deﬁne the concept of sufﬁcient representation for learning in an MDP.
Deﬁnition 3 (Sufﬁcient Representation for Learning) .
A representation mapping φ is sufﬁcient for a task M w.r.t.
approximation operator H φ if H φ Q π = Q π , for all π ∈ Π Dφ .
( φ is (cid:15) -sufﬁcient for learning if (cid:107)H φ Q π − Q π (cid:107) ≤ (cid:15) and linearly-sufﬁcient if H φ is linear transformation.)
Deﬁnition 3 suggests that the representation is sufﬁcient for learning a task as long as it is sufﬁcient for policies in Π Dφ .
Then, the lemma below justiﬁes that a nearly sufﬁcient representation can ensure that approximate policy iteration converges to a near-optimal solution.
Lemma 4 (Error Bound for Approximate Policy Iteration) .
If φ is (cid:15) -sufﬁcient for task M (with (cid:96) ∞ norm), then the approximated policy iteration with approximation operator H φ starting from any initial policy that is encoded by φ ( π 0 ∈
Π Dφ ) satisﬁes
[Equation]
where π k is the policy in the k -th iteration.
Lemma 4, proved in Appendix C, is extended from the error bound provided by Bertsekas & Tsitsiklis (1996).
For simplicity, we consider the bound in (cid:96) ∞ , but tighter bounds can be derived with other norms (Munos, 2005), although a tighter bound is not the focus of this paper.
How Can We Learn A Sufﬁcient Representation?
So far we have provided a principle to deﬁne whether a given representation is sufﬁcient for learning.
In DRL, the representation is learned together with the policy or value function using neural networks, but the quality of the representation may be poor (Dabney et al., 2020), which makes it hard for the agent to ﬁnd an optimal policy.
Based on Deﬁnition 3, a natural method to learn a good representation is to let the representation ﬁt as many policy values as possible as auxiliary tasks, which matches the ideas in other works.
For example, Bellemare et al.
(2019) propose to ﬁt a set of representative policies (called adversarial value functions ).
Dabney et al.
(2020) choose to ﬁt the values of all past policies (along the value improvement path ), which requires less computational resource. Different from these works that Require:
Regularization weight λ ; update frequency m for stable encoder.
[Table] [Table] [Table] [Table]
[Table]
[Table]
L P
L
Equation
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Equation] directly ﬁt the value functions of multiple policies, in Section 4.2, we propose a computationally efﬁcient way to realize sufﬁcient representation for learning and knowledge transferring: to ﬁt and transfer an auxiliary policy-independent dynamics model.
4. A LGORITHM : L EARNING AND T RANSFERRING M ODEL - BASED R EGULARIZER
Our goal is to use the knowledge learned in the source task to learn a good representation in the target task, such that the agent learns the target task more easily than learning from scratch.
Since we focus on developing a generic transfer mechanism, the base learner can be any DRL algorithms.
We use L base to denote the loss function of the base learner.
As motivated in Section 4.1, we propose to learn policy-independent dynamics models for producing high-quality representations: (1) ˆ P which predicts the representation of the next state for a given state action pair, and (2) ˆ R which predicts the immediate reward for a given state-action pair.
For a batch of N transition samples { o i , a i , o (cid:48)
i , r i } Ni =1 , deﬁne the transition loss and the reward loss as:
[Equation]
where ¯ φ ( o (cid:48) i ) denotes the representation of the next state o (cid:48)
i with stop gradients.
In order to ﬁt a more diverse state distribution, transition samples are drawn from an off-policy buffer, which stores shufﬂed past trajectories.
Note that a similar model loss is commonly used in model-based RL (van der
Pol et al., 2020) and bisimulation works Zhang et al.
(2020b),
but our focus is to use the learned model as an auxiliary task and to transfer the model to the target task.
[Figure]
The learning procedures for the source task and the target task are illustrated in Algorithm 1 and Algorithm 2, respectively.
Figure 2 depicts the architecture of the learning model for both source and target tasks.
z = φ ( o ) and z (cid:48)
=
¯ φ ( o (cid:48) ) are the encoded observation and next observation.
Given the current encoding z and the action a , the dynamics models ˆ P and ˆ R return the predicted next encoding ˆ z (cid:48)
=
ˆ P ( z, a ) and predicted reward ˆ r = ˆ R ( z, a ) .
Then the transition loss is the mean squared error (MSE) between z (cid:48) and ˆ z (cid:48) in a batch; the reward loss is the MSE between r and ˆ r in a batch.
In the source task (Algorithm 1): dynamics models ˆ P and ˆ R are learned by minimizing L P and
L R , which are computed based on a recent copy of encoder called stable encoder ˆ φ ( S ) (Line 5).
The computation of the stable encoder is to help the dynamics models converge, as the actual encoder φ ( S ) changes at every step.
Note that a stable copy of the network is widely used in many DRL algorithms (e.g. the target network in DQN), which can be directly regarded as ˆ φ ( S ) without maintaining an extra network.
The actual encoder φ ( S ) is regularized by the auxiliary dynamics models ˆ P and ˆ R (Line 6).
In the target task (Algorithm 2): dynamics model ˆ P and ˆ R are transferred from the source task and ﬁxed during learning.
Therefore, the learning of φ ( T ) is regularized by static dynamics models, which leads to faster and more stable convergence than naively learning an auxiliary task.
Relation and Difference with Model-based RL and Bisimulation Metrics.
The goal of modelbased RL (Kipf et al., 2019; Grimm et al., 2020) is to learn an accurate world model and use the model for planning.
The dynamics model could be learned on either raw observations or representations.
In our framework, we also learn a dynamics model, but the model serves as an auxiliary task, and learning is still performed by the model-free base learner with L base .
Bisimulation metrics (Castro, 2020; Zhang et al., 2020b) aim to approximate the distances among states by learning dynamics models, whereas we do not explicitly measure the distance among states.
Note that we also do not require a reconstruction loss that is common in literature (Lee et al., 2019).
4. T HEORETICAL A NALYSIS : B ENEFITS OF T RANSFERABLE D YNAMICS M ODEL
The algorithms introduced in Section 4.2 consist of two designs: learning a latent dynamics model as an auxiliary task, and transferring the dynamics model to the target task.
In this section, we show theoretical justiﬁcations and practical advantages of our proposed method.
We aim to answer the following two questions: (1) How does learning an auxiliary dynamics model help with representation learning?
(2) Is the auxiliary dynamics model transferable?
For notational simplicity, let P a and R a denote the transition and reward functions associated with action a ∈ A .
Note that P a
and R a are independent of any policy.
We then deﬁne the sufﬁciency of a representation mapping w.r.t.
dynamics models as below.
Deﬁnition 5 (Policy-independent Model Sufﬁciency) .
For an MDP M , a representation mapping φ is sufﬁcient for its dynamics ( P a , R a ) a ∈A if ∀ a ∈ A , there exists functions ˆ P a : R d → R d and ˆ R a : R d → R such that ∀ o ∈
O , ˆ P a ( φ ( o ))
=
E o (cid:48) ∼ P a ( o ) [ φ ( o (cid:48) )] , ˆ R a ( φ ( o ))
=
R a ( o ) .
Remarks.
(1) Since we focus on learning a deterministic transition prediction model ˆ P , φ is exactly sufﬁcient for dynamics ( P a , R a ) a ∈A only when the transition function P is deterministic.
If P is stochastic, but we have max o,a (cid:107) E o (cid:48) ∼ P a ( o ) [ φ ( o (cid:48) )] − ˆ P a ( φ ( o ))
(cid:107) ≤ (cid:15) P and max o,a | R a ( o ) − ˆ R a ( φ ( o ))
| ≤ (cid:15) R , then φ is ( (cid:15) P , (cid:15) R ) -sufﬁcient for the dynamics of M .
Next we show by Proposition 6 and Theorem 7 that learning sufﬁciency can be achieved via ensuring model sufﬁciency.
Proposition 6 (Learning Sufﬁciency Induced by Policy-independent Model Sufﬁciency) .
Consider an MDP M with deterministic transition function P and reward function R .
If φ is sufﬁcient for ( P a , R a ) a ∈A , then it is sufﬁcient (but not necessarily linearly sufﬁcient) for learning in M .
Proposition 6 shows that, if the transition is deterministic and the model errors L P , L R are zero, then φ is exactly sufﬁcient for learning.
More generally, if the transition function P is not deterministic, and model ﬁtting is not perfect, the learned representation can still be nearly sufﬁcient for learning as characterized by Theorem 7 below, which is extended from a variant of the value difference bound derived by Gelada et al.
Proposition 6 and Theorem 7 justify that learning the latent dynamics model as an auxiliary task encourages the representation to be sufﬁcient for learning. The model error L P and L R deﬁned in Section 4.2 can indicate how good the representation is.
Theorem 7.
For an MDP M , if representation mapping φ is ( (cid:15) P , (cid:15) R ) -sufﬁcient for the dynamics of M , then approximate policy iteration with approximation operator H φ starting from any initial policy π 0 ∈ Π
Dφ satisﬁes
[Equation]
where K φ,V is an upper bound of the value Lipschitz constant as deﬁned in Appendix B.
Transferring Model to Get Better Representation in Target.
Although Proposition 6 shows that learning auxiliary dynamics models beneﬁts representation learning, ﬁnding the optimal solution is non-trivial since one still has to learn ˆ P and ˆ R .
Therefore, the main idea of our algorithm is to transfer the dynamics models ˆ P, ˆ R from the source task to the target task, to signiﬁcantly ease the learning in the target task.
Theorem 8 below guarantees that transferring the dynamics models is feasible.
Our experimental result in Section 6 veriﬁes that learning with transferred and ﬁxed dynamics models outperforms learning with randomly initialized dynamics models.
Theorem 8 (Transferable Dynamics Models) .
Consider a source task M ( S ) and a target task M ( T ) with deterministic transition functions.
Suppose φ ( S ) is sufﬁcient for ( P ( S ) a , R ( S ) a ) a ∈A with functions
ˆ P a , ˆ R a , then there exists a representation φ ( T ) satisfying ˆ P a ( φ ( o ))
=
E o (cid:48) ∼ P ( T ) a [ φ ( o (cid:48) )] , ˆ R a ( φ ( o )) =
R ( T ) a ( o ) , for all o ∈ O ( T ) , and φ ( T ) is sufﬁcient for learning in M ( T ) .
Theorem 8 shows that the learned latent dynamics models ˆ P, ˆ R are transferable from the source task to the target task.
For simplicity, Theorem 8 focuses on exact sufﬁciency as in Proposition 6, but it can be easily extended to (cid:15) -sufﬁciency if combined with Theorem 7.
Proofs for Proposition 6,
Theorem 7 and Theorem 8 are all provided in Appendix C.
Trade-off between Approximation Complexity and Representation Complexity. As suggested by Proposition 6, ﬁtting policy-independent dynamics encourages the representation to be sufﬁcient for learning, but not necessarily linearly sufﬁcient.
Therefore, we suggest using a non-linear policy/value head following the representation to reduce the approximation error.
Linear sufﬁciency can be achieved if φ is made linearly sufﬁcient for P π and R π for all π ∈ Π Dφ , where P π and R π are transition and reward functions induced by policy π (Proposition 10, Appendix B).
However, using this method for transfer learning is expensive in terms of both computation and memory, as it requires to learn P π and R π for many different π ’s and store these models for transferring to the target task.
Therefore, there is a trade-off between approximation complexity and representation complexity.
Learning a linearly sufﬁcient representation reduces the complexity of the approximation operator.
But it requires more complexity in the representation itself as it has to satisfy much more constraints.
To develop a practical and efﬁcient transfer method, we use a slightly more complex approximation operator (non-linear policy head) while keeping the auxiliary task simple and easy to transfer across tasks.
Please see Appendix B for more detailed discussion about linear sufﬁciency.
R ELATED W ORK
Transfer RL across Observation Feature Spaces. Transferring knowledge between tasks with different observation spaces has been studied for years.
Many existing approaches(Taylor et al., 2007; Mann & Choe, 2013;
Brys et al., 2015) require an explicit mapping between the source and target observation spaces, which may be hard to obtain in practice.
Raiman et al.
(2019) introduce network surgery that deals with the change in the input features by determining which components of a neural network model should be transferred and which require retraining.
However, it requires knowledge of the input feature maps, and is not designed for drastic changes, e.g. vector to pixel.
Sun et al.
(2020) propose a provably sample-efﬁcient transfer learning algorithm that works for different observation spaces without knowing any inter-task mapping, but the algorithm is mainly designed for tabular RL and model-based RL which uses the model to plan for a policy, different from our setting.
Gupta et al.
(2017) achieve transfer learning between two different tasks by learning an invariant feature space, with a key time-based alignment assumption. We empirically compared this method with our proposed transfer algorithm in Section 6. Our work is also related to state abstraction in block MDPs, as studied by Zhang et al. (2020a).
But the problem studied in Zhang et al. (2020a) is a multi-task setting where the agent aims to learn generalizable abstract states from a series of tasks. Another related topic is domain adaptation in RL (Higgins et al., 2017; Eysenbach et al., 2020; Zhang et al., 2020b), where the target observation space (e.g. real world) is different from the source observation (e.g. simulator).
However, domain adaptation does not assume drastic observation changes (e.g. changed dimension).
Moreover, the aim of domain adaptation is usually zero-shot generalization to new observations, thus prior knowledge or a few samples of the target domain is often needed (Eysenbach et al., 2020).
Representation Learning in RL. In environments with rich observations, representation learning is crucial for the efﬁciency of RL methods.
Learning unsupervised auxiliary tasks (Jaderberg et al., 2016) is shown to be effective for learning a good representation.
The relationship between learning policy-dependent auxiliary tasks and learning good representations has been studied in some prior works (Bellemare et al., 2019; Dabney et al., 2020; Lyle et al., 2021), while our focus is to learn policy-independent auxiliary tasks to facilitate transfer learning.
Using latent prediction models to regularize representation has been shown to be effective for various types of rich observations (Guo et al., 2020; Lee et al., 2019).
Gelada et al.
(2019) theoretically justify that learning latent dynamics model guarantees the quality of the learned representation, while we further characterize the relationship between representation and learning performance, and we utilize dynamics models to improve transfer learning.
Zhang et al.
(2020b) use a bisimulation metric to learn latent representations that are invariant to task-irrelevant details in observation.
As pointed out by Achille & Soatto (2018), invariant and sufﬁcient representation is indeed minimal sufﬁcient, so it is an interesting future direction to combine our method with bisimulation metric to learn minimal sufﬁcient representations.
There is also a line of work using contrastive learning to train an encoder for pixel observations (Srinivas et al., 2020; Yarats et al., 2021; Stooke et al., 2021), which usually pre-train an encoder based on image samples using self-supervised learning.
However, environment dynamics are usually not considered during pre-training.
Our algorithm can be combined with these contrastive learning approaches to further improve learning performance in the target task.
E XPERIMENTAL E VALUATION
We empirically evaluate our transfer learning algorithm in various environments and multiple observation-change scenarios.
Detailed experiment setup and hyperparameters are in Appendix E. Baselines.
To verify the effectiveness of our proposed transfer learning method, we compare our transfer learning algorithm with 4 baselines: (1) Single : a single-task base learner.
(2) Auxiliary : learns auxiliary models from scratch to regularize representation. (3) Fine-tune : loads and freezes the source policy head, and retrains an encoder in the target task. (4) Time-aligned (Gupta et al., 2017): supposes the target task and the source task proceed to the same latent state given the same action sequence, and pre-trains a target-task encoder with saved source-task trajectories. More details of baseline implementations are in Appendix E.1.1.
Scenarios. As motivated in Section 1, there are many scenarios where one can beneﬁt from transfer learning across observation feature spaces. We evaluate our proposed transfer algorithm in 7 environments that ﬁt various scenarios, to simulate real-world applications: (1) Vec-to-pixel: a novel and challenging scenario, where the source task has low-dimensional vector observations and the target task has pixel observations.
We use 3 vector-input environments CartPole, Acrobot and Cheetah-Run as source tasks, and use the rendered image in the target task. (2) More-sensor: another challenging scenario where the target task has a lot more sensors than the source task. We use 3 MuJoCo environments: HalfCheetah, Hopper and Walker2d, whose original observation dimensions are 17, 11 and 17, respectively.
We add mass-based inertia and velocity (provided by MuJoCo’s API), resulting in 145, 91, 145 dimensions in the corresponding target tasks. (3) Broken-sensor: we use an existing game 3DBall contained in the Unity ML-Agents Toolkit (Juliani et al., 2018), which has two different observation speciﬁcations that naturally ﬁt our transfer setting: the source observation has 8 features containing the velocity of the ball; the target observation does not have the ball’s velocity, thus the agent has to stack the past 9 frames to infer the velocity. Please see Appendix E.1.2 for more detailed descriptions of all the 7 environments. Base DRL Learners.
What we propose is a transfer learning mechanism that can be combined with any existing DRL methods. For environments with discrete action spaces (CartPole, Acrobot), we use the DQN algorithm (Mnih et al., 2015), while for environments with continuous action spaces (Cheetah-Run, HalfCheetah, Hopper, Walker2d, 3DBall), we use the SAC algorithm (Haarnoja et al., 2018). To ensure a fair comparison, we use the same base DRL learner with the same hyperparameter settings for all tested methods, as detailed in Appendix E.1.3.
As is common in prior works, our implementation of the RL algorithms is mostly a proof of concept, thus many advanced training techniques are not included (e.g. Rainbow DQN). Results.
Experimental results on all tested environments are shown in Figure 3. We can see that our proposed transfer method learns signiﬁcantly better than the single-task learner, and also outperforms all baselines in the challenging target tasks.
Our transfer method outperforms Auxiliary since it transfers dynamics model from the source task instead of learning it from scratch, and outperforms Fine-tine since it regularizes the challenging encoder learning with a model-based regularizer. [Figure] The Time-aligned method, although requires additional pre-training that is not shown in the ﬁgures, does not work better than Single in most environments, because the time-based alignment assumption may not hold as discussed in Appendix E.1.1.
In some environments (e.g. Hopper, Walker2d, 3DBall), our transfer algorithm even achieves better asymptotic performance than the source-task policy, which suggests that our method can be used for improving the policy with incremental observation design. To the best of our knowledge, we are the ﬁrst to achieve effective knowledge transfer from a vector-input environment to a pixel-input environment without any pre-deﬁned mappings.
[Figure]
Ablation Study and Hyper-parameter Test. To verify the effectiveness of proposed transfer method, we conduct ablation study and compare our method with its two variants: only transferring the transition model ˆ P and only transferring the reward model ˆ R .
Figure 4 shows the comparison in HalfCheetah, and Appendix E.2 demonstrates more results.
We ﬁnd that all the variants of our method can make some improvements, which suggests that transferring ˆ P and ˆ R are both effective designs for accelerating the target task learning.
Figurein Appendix E.2 shows another ablation study where we investigate different selections of model regularizers and policy heads.
In Algorithm 2, a hyper-parameter λ is needed to control the weight of the transferred model-based regularizer.
Figure 7 in Appendix E.2 shows that, for a wide range of λ ’s, the agent consistently outperforms the single-task learner.
Potential Limitations and Solutions.
As Figure 3 shows, in some environments such as HalfCheetah, our transfer algorithm signiﬁcantly outperforms baselines without transfer.
But in Walker2d, the improvement is less signiﬁcant, although transferring is still better than not transferring.
This phenomenon is common in model-based learning (Nagabandi et al., 2018), as state predicting in Walker2d is harder than that in HalfCheetah due to the complexity of the dynamics.
Therefore, we suggest using our method to transfer when the learned models ( ˆ P, ˆ R ) in the source task are relatively good (error is low).
More techniques of improving model-based learning, such as bisimulation (Zhang et al., 2020b; Castro, 2020), can be applied to further improve the transfer performance.
C ONCLUSION
In this paper, we identify and propose a solution to an important but rarely studied problem: transferring knowledge between tasks with drastically different observation spaces where inter-task mappings are not available.
We propose to learn a latent dynamics model in the source task and transfer the model to the target task to facilitate representation learning.
Theoretical analysis and empirical study justify the effectiveness of the proposed algorithm.
E THICS S TATEMENT
Transfer learning aims to apply previously learned experience to new tasks to improve learning efﬁciency, which is becoming more and more important nowadays for training intelligent agents in complex systems.
This paper focuses on a practical but rarely studied transfer learning scenario, where the observation feature space of an RL environment is subject to drastic changes.
Driven by theoretical analysis on representation learning and its relation to latent dynamics learning, we propose a novel algorithm that transfers knowledge between tasks with totally different observation spaces, without any prior knowledge of an inter-task mapping.
This work can beneﬁt many applications as suggested by the examples below.
(1) In many real-life environments where deep RL is used (e.g. navigating in a building), the underlying dynamics (e.g. the structure of the building) are usually ﬁxed, but what features the observation space has is designed by human developers (e.g. what sensors are installed) and thus may change frequently during the development.
When the agent gets equipped with better sensors, our algorithm makes it possible to reuse previously learned models when learning with the new sensors.
(2) An agent usually learns better with a compact observation space (e.g. a low-dimensional vector space containing its location and the goal’s location) than a rich/noisy observation space (e.g. an image containing the goal).
However, a compact observation is usually more difﬁcult to construct in practice as it may require expert knowledge and human work.
In this case, one can extract compact observations in a few samples and pre-train a policy with our Algorithm 1, then train the agent in the real environment with rich observations with our Algorithm 2 using the learned dynamics models.
Our experiment in Figure 3 shows that the learning efﬁciency in the rich-observation environment can be signiﬁcantly improved with our proposed transfer method.
R EPRODUCIBILITY S TATEMENT
For theoretical results, we provide concrete proofs in Appendix C and Appendix D. For empirical results, we illustrate implementation details in Appendix E.
The source code and running instructions are provided in the supplementary materials.
Appendix: Transfer RL across Observation Feature Spaces via Model-Based Regularization A A DDITIONAL P RELIMINARY K NOWLEDGE
For any policy π , its Q value Q π is the unique ﬁxed point of the Bellman operator [Equation] The optimal policy can be found by policy iteration (Howard, 1960), where one starts from an initial policy π 0 and repeats policy evaluation and policy improvement. More speciﬁcally, at iteration k , the algorithm evaluates Q π k via Equation (4), then improves the policy by π k +1 ( o ) :
= argmax a ∈A Q π k ( o, a ) , ∀ o ∈ O .
It is well-known that the policy iteration algorithm converges to the optimal policy under mild conditions (Puterman, 2014).
When the dynamics P and R are unknown, reinforcement learning algorithms use interaction samples from the environment to approximately solve ˆ Q π k .
Prior works (Bertsekas & Tsitsiklis, 1996; Munos, 2005) have shown that if the approximation error is bounded by a small constant, the performance of π k as k → ∞ is guaranteed to be close to the optimal policy value Q π ∗ , which we also denote as Q ∗ .
B A DDITIONAL D ISCUSSION FOR R EPRESENTATION S UFFICIENCY
Good Representation for A Fixed Policy The following deﬁnition characterizes the sufﬁciency of a representation mapping in terms of evaluating a ﬁxed policy.
Deﬁnition 9 (Sufﬁcient Representation for A Fixed Policy) .
A representation mapping φ is sufﬁcient for a policy π w.r.t.
approximation operator H φ
iff H φ V π
= V π .
More generally, for a constant (cid:15) ≥ 0 , Φ is (cid:15) -sufﬁcient for π iff (cid:107)H φ V π − V π (cid:107) ≤ (cid:15) .
Remarks.
(1) If o 1 , o 2 ∈ O have different values under π , a good representation should be able to distinguish them, i.e., φ ∗ ( o 1 ) (cid:54) =
φ ∗ ( o 2 ) .
(2) The approximation operator H φ is linear if H φ V =
Proj
Φ ( V ) where Proj Φ ( · ) denotes the orthogonal projection to the subspace spanned by the basis functions of (cid:104) φ 1 , φ 2 , · · · , φ d (cid:105) .
Model Sufﬁciency over Policies Induces Linear Learning Sufﬁciency It can be found from Deﬁnition 3 that φ is sufﬁcient as long as it represents Q π for all π ∈ Π Dφ .
Fitting various value functions to improve representation quality is proposed by some prior works Bellemare et al.
; Dabney et al.
and shown to be effective.
However, learning and ﬁtting many policy values could be computationally expensive, and is not easy to be applied to transfer learning between tasks with different observation spaces.
Can we regularize the representation with the latent dynamics instead of policy values?
Proposition 10 below shows that if Φ is linearly sufﬁcient for all dynamics pairs ( P π , R π ) induced by policies in Π Dφ and the dynamics pairs associated with all actions, then Φ is linearly sufﬁcient for learning.
For notation simplicity, assume the state space is ﬁnite.
Then, let ( P π , R π ) be the transition matrix and reward vector induced by policy π ,
i.e., P π
[ i, j ] = E a ∼ π
( o i )
[ P ( o j |
o
i , a )] , R π
[ i ] =
E a ∼ π
( o i )
[ R ( o i , a )] .
Similarly, let P a [ i, j ] =
P
o j | o i , a ) , R a
[ i ] =
R ( o i , a ) .
We let Φ denote the representation matrix, where the i -th row of Φ refers to the feature of the i -the observation.
Proposition 10 (Linear Sufﬁciency Induced by Policy-based Model Sufﬁciency) .
For representation Φ , if for all π ∈ Π Dφ , a ∈ A , there exist ˆ P π , ˆ R π , ˆ P a , ˆ R a such that Φ ˆ P π
= P π Φ , Φ ˆ R π = R π , Φ ˆ P a = P a Φ , Φ ˆ R a = R a , i.e. Φ is linearly sufﬁcient both policy-based dynamics and policyindependent dynamics models, i.e., then Φ is linearly sufﬁcient for a task M w.r.t.
approximation operator H φ .
Proposition 10 proven in Appendix C.3 suggests that we can let representation ﬁt ( P π , R π ) for many different π ’s.
However, it could be computationally intractable since the policy space is large.
More importantly, it is hard to store and transfer a large number of dynamics models for all ( ˆ P π , ˆ R π ) .
In our Proposition 6, we show that learning sufﬁciency can be induced by policy-independent model sufﬁciency, which is much simpler as there is no need to learn and store ( ˆ P π , ˆ R π ) for many policies.
As a trade-off, the policy-independent model induces non-linear sufﬁciency instead of linear sufﬁciency, requiring a more expressive approximation operator.
Latent MDP induced by Representation
We follow the analysis by Gelada et al.
and deﬁne a new MDP under the representation mapping φ :
˜ M = (cid:104)
˜ O , A , ˜ P, ˜ R, γ (cid:105) , where for all o ∈ O , φ ( o ) ∈
˜ O , ˜ P
φ ( o ) , a )
=
ˆ P a φ ( o ) , ˜ R ( φ ( o ) , a )
=
ˆ R a φ ( o ) .
Let ˜ V denote the value function in ˜ M , and let ˜ π denote a policy in ˜ M .
We make the following mild assumption
Assumption 11.
There exists a constant K φ,V such that
[Equation]
This assumption is mild as any MDP with bounded reward has bounded value functions.
C T ECHNICAL P ROOFS C. P ROOF OF L EMMA
Proof of Lemma 4.
We ﬁrst show that policy iteration with approximation operator H φ starting from a policy π 0 ∈ Π
Dφ generates a sequence of policies that are in Π Dφ .
That is, for all π k , and any o 1 , o 2 ∈ O , π k ( o 1 ) =
π k ( o 2 ) if φ ( o 1 ) =
φ ( o 2 ) .
We prove this claim by induction.
Base case: when k = 0 , π 0 ∈ Π Dφ .
Inductive step: assume π k ∈ Π
Dφ for k ≥ 0 , then for iteration k + 1 , we know that
[Equation]
(5) where Q k = H φ Q π k .
Based on the deﬁnition of H φ , Q k ( o, a ) =
f ( φ ( o ); θ a ) for some θ a .
Hence, if o 1 and o 2 have the same representation, Q k ( o 1 , · ) and Q k ( o 2 , · ) are equal.
As a result, π k +1 ( o 1 ) and π k +1 ( o 2 ) are equal, so π k +1 ∈ Π Dφ .
Next we prove the error bound in Lemma 4.
We start by restating the error bounds for approximate policy iteration by Bertsekas & Tsitsiklis (1996):
[Equation]
where π k is the policy in the k -th iteration.
Then we extend the above result to the action value Q .
For any π k during the policy iteration (as proven above, π k ∈ Π Dφ ), if φ is (cid:15) -sufﬁcient for M as deﬁned in Deﬁnition 3 with (cid:96) ∞ norm, then we have (cid:107) Q
k − Q π k (cid:107) ∞ ≤ (cid:15) .
That is, ∀ o ∈ O , a ∈
A , | Q [Equation] o, a ) − Q π k ( o, a ) | ≤ (cid:15) .
Therefore, ∀ o [Equation] O ,
[Equation] [Equation]
On the other hand, we can derive
[Equation]
Combining Equation (7) and (10) [Equation] we obtain
[Equation]
C. P ROOF OF P ROPOSITION
Proof of Proposition 6.
Given that P is deterministic for o ∈ O , a ∈ A , we slightly abuse notation and let o (cid:48)
=
P ( o, a ) =
P a ( o ) if P ( o (cid:48) | o, a ) = 1 .
If φ is sufﬁcient for the dynamics models, i.e. ∀ o ∈
O , a ∈ A , ˆ P a φ ( o ) =
φ ( P a ( o )) , ˆ R a φ ( o )
=
R a ( o ) .
Then, we can deﬁne a new MDP ˜ M = (cid:104) ˜
O , A , ˜ P, ˜ R, γ (cid:105) , where for all o ∈ O , φ ( o ) ∈
˜ O , and ˜ P ( φ ( o ) , a )
=
ˆ P a φ ( o ) =
φ ( P ( o, a )) , ˜ R ( φ ( o ) , a ) =
ˆ R a φ ( o ) =
R ( o, a ) .
Any policy π ∈ Π Dφ , based on the deﬁnition of Π Dφ , can be written as ˜ π ◦ φ , where ˜ π is a deterministic policy in ˜ M .
Next, we show that for all o ∈ O , a ∈ A , Q π ( o ) =
˜ Q ˜ π ( φ ( o )) .
By deﬁnition of the Q value, we know
[Equation]
[Equation] We claim that in the above equations, ˜ o t = φ ( o t ) , ˜ a t = a t , for all t ≥ 0 .
We prove the claim by induction.
When t = 0 , the claim holds as ˜ o
= φ ( o 0 ) =
φ ( o ) , ˜ a 0 = a
= a .
Then, with inductive hypothesis that ˜ o t = φ ( o t ) , ˜ a t = a t , we show the claim holds for t + 1 :
Action: a t +1 =
π ( o t +1 ) =
˜ π ( φ ( o t +1 ))
=
˜ a t +1 .
State:
˜ o t +1 =
˜ P
( ˜ o t , a t ) =
˜ P ( φ ( o t ) , a t ) = φ ( P ( o t , a t )) =
φ ( o t +1 ) .
Hence, we have shown ˜ o t = φ ( o t ) , ˜ a t = a t , for all t ≥ 0 , then the reward in the t -th step of
Equation (12) and (13) are the same, as ˜ R
( ˜ o t , ˜ a t ) =
˜ R ( φ ( o t ) , a t ) =
R ( o t , a t ) .
Therefore, Q π ( o ) = ˜
Q ˜ π ( φ ( o )) .
Therefore, for any π ∈ Π Dφ , its action value can be represented by ˜ Q ˜ π ◦ φ .
Therefore, φ is sufﬁcient for learning in M .
Next, we see whether φ is linearly sufﬁcient for learning the task.
Consider an arbitrary policy π ∈ Π Dφ . Without loss of generality, suppose R π is linearly represented by φ , i.e. R a ( o, a ) = φ ( o ) (cid:62) ˆ R a , then we have [Equation]
where ˆ R :=
[ ˆ R a 1 ; ˆ R a 1 ; · · · ; ˆ R a |A| ] .
We can ﬁnd that unless π always takes the same action for all input states, φ is not guaranteed to linearly encode R π .
Similarly, for P π , suppose P a ( ·| o, a ) =
φ ( o ) (cid:62) ˆ P a we have
[Equation]
where ˆ P :=
[ ˆ P a 1 ; ˆ P a 1 ; · · · ; ˆ P a |A| ] is an |A| ×
d × d tensor, and ˆ P ( · , · , · ) denotes the multi-linear operation.
Hence, if π takes different actions in different states, φ may not linearly encode the transition, either.
Therefore, φ is not guaranteed to linearly encode R π and P π , and thus is not guaranteed to linearly encode V π and Q π .
C. P ROOF OF P ROPOSITION
Proof of Proposition 10.
We ﬁrst show that for any π ∈ Π , if φ is linearly sufﬁcient for ( P π , R π ) , then there exists a vector ω ∈ R k such that V
π
=
ˆ V π
= Φ ω .
Since Φ is linearly sufﬁcient for P π and R π , we have Φ ˆ P π
= P π Φ and Φ ˆ R π
= R π for some ˆ P π and ˆ R π .
Let ω
= ( I − γ ˆ P π ) † ˆ R π , then the Bellman error of ˆ V π = Φ ω can be computed as
[Equation]
Therefore, ˆ V π is a ﬁxed point of the Bellman operator T π , and thus equal to V π .
Next, as we know that Q π ( · , a ) = R a + γ (cid:104) P a , V π (cid:105) , and Φ ˆ P a = P a Φ , Φ ˆ R a
= R a , we can obtain
[Equation]
Therefore, for any π ∈ Π Dφ , Q π can be represented by a linear approximation operator on Φ , and thus Φ is linearly sufﬁcient for learning.
C. P ROOF OF T HEOREM
Proof of Theorem 7.
Lemma 2 in Gelada et al.
(2019) is based on one policy in the induced MDP and bounded model errors.
We can replace the Wasserstein distance W ( φP ( ·| o, a ) , ˜ P ( ·| φ ( o ) , a )) by the Euclidean distance (cid:107) φP ( o, a ) ,
˜ P ( φ ( o ) , a ) (cid:107) as we focus on deterministic transitions.
For any policy π ∈ Π Dφ that can be written as ˜ π ◦ φ , we have
[Equation]
Therefore, φ is (1 − γ ) − 1 ( (cid:15) R + γK φ,V (cid:15) P ) -sufﬁcient for learning M .
Combined with Lemma 4, we can obtain the bound in Theorem 7.
C. P ROOF OF T HEOREM
Proof of Theorem 8.
First of all, if there exists φ ( T ) satisfying ˆ P a ( φ ( o i ))
=
P ( T ) a
[ i ]Φ ( T ) , ˆ R a ( φ ( o i ))
=
R ( T ) a [ i ] , ∀ o i ∈
O ( T ) , then it is sufﬁcient for the dynamics of the target task, and thus sufﬁcient for learning the target task as stated in Proposition 6.
Therefore, our focus is to show the existence of such a representation.
As φ ( S ) is sufﬁcient for P ( S ) a and R ( S ) a for all a ∈ A , we have
[Equation]
) where we let P ( S ) ( o ( S ) , a ) denote the next state of ( o ( S ) , a ) , given that P is deterministic.
Based on Assumption 1, we know that there exists a function f
such that ∀ o ( T ) ∈
O ( T ) , f
( o ( T ) ) ∈
O ( S ) , and f ( P ( T ) ( o ( T ) , a ))
=
P ( S ) ( f ( o ( T ) ) , a ) , R ( T ) ( o ( T ) , a ) =
R ( S )
( f ( o ( T ) ) , a ) .
Hence, we can obtain
[Equation] [Equation]
Let ˆ φ ( T ) :
= φ ( S ) ◦ f , then we get [Equation]
[Equation]
Therefore, ˆ φ ( T ) is a feasible solution satisfying model sufﬁciency in the target task, and thus is sufﬁcient for learning.
Theorem 8 holds since we have shown (1) all feasible solutions to Φ ( T ) ˆ P a = P ( T ) a Φ and Φ ( T ) ˆ R a =
R ( T ) a are sufﬁcient for learning in M ( T ) , and (2) there exists at least one feasible solution to Φ ( T )
ˆ P a =
P ( T ) a Φ and Φ ( T ) ˆ R a = R ( T ) a .
D R EPRESENTATION L EARNING FOR A PPROXIMATED V ALUE I TERATION
Now we illustrate how our transfer algorithm and the proposed model-based regularization work for approximated value iteration.
We focus on the case where the reward function R ( o, a ) ≥ 0 for all o ∈ O and a ∈ A .
Preliminaries
The bases of value iteration is the Bellman optimality operator T ∗ .
For the value function, we have
[Equation] [Equation]
For the Q function, we have
[Equation] [Equation]
where we slightly abuse notation and use T ∗ for both the value function and the Q function when there is no ambiguity.
Starting from some initial Q 0 (or V 0 ) and iteratively applying T ∗ , i.e., Q k +1 =
T ∗ Q k , Q k (or
V k ) can ﬁnally converge to Q ∗ or V ∗ when k → ∞ , which is known as value iteration.
When an approximation operator H is used, the process is called approximate value iteration (AVI): Q k +1 =
H φ T ∗ Q k .
A prior work has shown the following asymptotic result:
Lemma 12 (Approximate Value Iteration for Q ) .
For a reinforcement learning algorithm based on value iteration with approximation operator H , if (cid:107)HT ∗ Q k − T ∗ Q k (cid:107) ∞ ≤ (cid:15) , for all Q k along the value iteration path, then we have
[Equation]
Value Iteration Learning with Given Representation Given a representation mapping φ , we aim to learn an approximation function h : φ ( O ) → R | A | such that ˆ Q ( o, · )
=
h ( φ ( o ))
≈ Q ( o, · ) .
For notation simplicity, we further use h ( φ ( o ) , a ) to denote the approximated Q value ˆ Q ( o, a ) for a ∈ A .
We start from an initial h 0 that has a uniform value c for all inputs, where c > 0 can be randomly selected.
The initial approximation for Q value is ˆ Q
=
h 0 ◦ φ .
Then, at iteration k > 0 , we solve ˆ Q k
= H φ T ∗ ˆ Q k − 1 =
h k ◦ φ , where h k :=
argmin h (cid:107) h ◦ φ − T ∗ ˆ Q k − 1 (cid:107) ∞ .
We use a neural network (universal function approximator) to parameterize h , so the approximation error (cid:107) h ◦ φ − T ∗ ˆ Q k − 1 (cid:107) ∞ depends on the representation quality of φ .
Therefore, a representation mapping φ is (cid:15) -sufﬁcient for learning with value iteration if (cid:107)H φ T ∗ Q k −
T ∗ Q k (cid:107) ∞ ≤ (cid:15) .
Next, we identify the relationship between policy-independent model sufﬁciency and the learning sufﬁciency with value iteration methods.
Guaranteed Learning with Model-regularized Representation
We ﬁrst make the following assumption for the learned approximation function h .
Assumption 13 (Lipschitz Value Approximation) .
There exists a constant K φ,h , such that ∀ k ≥ 0 , [Equation] [Equation]
[Equation]
24) where h k is the approximation function in the k -th iteration.
Then, the following Theorem holds, which justiﬁes that learning with model-regularized representation helps with value iteration learning.
Theorem 14.
For an MDP M , if encoder φ satisﬁes max o ∈O ,a ∈A | R ( o, a ) − ˆ R a ( φ ( o )) | ≤ (cid:15) R and max o ∈O ,a ∈A (cid:107) E o (cid:48) ∼ P ( ·| o,a ) φ ( o (cid:48) ) − ˆ P a ( φ ( o ))
(cid:107) 2 ≤ (cid:15) P for dynamics models ( ˆ P a , ˆ R a ) a ∈A , then the approximated value iteration with approximation operator H φ under Assumption 13 satisﬁes
[Equation]
Proof of Theorem 14.
Let h k be the approximation function in the k -th iteration.
That is, the approximated Q function in the k -th iteration is ˆ Q k = h k ◦ φ .
As the rewards of all state-action pairs are non-negative, we have h k ( φ ( o ) , a ) ≥ 0 .
Deﬁne a function ˆ h k
[Equation]
1 as
[Equation]
Given that
[Equation]
28) we have that for any o ∈ O , a ∈ A ,
[Equation]
29) =
[Equation]
[Equation]
where (33) is due to the non-negativity of h k , and (34) is due to Assumption 13.
Now we have shown that the constructed ˆ h k +1 satisﬁes [Equation]
[Equation]
According to the deﬁnition of H φ , we obtain
[Equation] [Equation]
since H φ ﬁnds a h k +1 that minimizes the approximation error.
Therefore, Theorem 14 follows by combining Inequality (37) and Lemma
E E XPERIMENT D ETAILS AND A DDITIONAL R ESULTS E. E XPERIMENT S ETTING D ETAILS E.1. B ASELINES
• Single : A DQN or SAC learner on the target domain without any auxiliary tasks.
• Auxiliary : On the target domain, the encoder φ ( T ) is optimized based on the loss L base ( φ ( T ) , π ( T ) )
+ λ (cid:104)
L P ( φ ( T ) ; ˆ P ( T ) )
+ L R ( φ ( T ) ; ˆ R ( T ) ) (cid:105) . Compared with our transfer algorithms which transfer the learned dynamics from source domain to the target domain, it learns the dynamics model ( ˆ P ( T ) , ˆ R ( T ) ) on the target domain from scratch. Here we set λ to be the same as our transferred algorithm (values of λ are provided in Appendix E.1.3).
The purpose of this baseline is to test whether the efﬁciency of our proposed transfer algorithms come from the transferred latent dynamics or from the auxiliary loss (or potentially both).
• Fine-tune : To test whether our transfer algorithms beneﬁt from loading the learned policy head π ( S ) , on the target domain, we load the weights of π ( T ) from the trained source policy head π ( S ) and train the DQN or SAC agent without any auxiliary loss.
• Time-aligned : Gupta et al.
(2017) propose to learn aligned representations for two tasks, under the assumption that the source-task agent and the target-task agent reach similar latent sates at the same time step, i.e. φ ( T ) ( s ( T ) t ) =
φ ( S ) ( s ( S ) t ) .
Note that this assumption is valid when the initial state is ﬁxed and the transitions are all deterministic.
Although in our setting, the agent can not learn both tasks simultaneously, we can adapt the idea of time-based alignment and encourage the target encoder
to map target observations to the source representations happening at the same time step.
In our experiments, we store N source trajectories (cid:110)
s i 0 , a i 0 , s i 1 , a i 1 , ..., (cid:111) N i =1 collected during source task training.
Then on the target domain, we ﬁrst collect N trajectories following the same action as the one collected from the source domain.
In other words, at time step t of the i -th trajectory, we take action a it .
After the target trajectories are collected, we minimize the alignment loss L align ( φ ( T ) )
=
{+E (cid:104)(cid:0) φ ( T ) ( s ( T ) t ) − φ ( S ) ( s ( S ) t ) (cid:1) 
+}
(cid:105) to enforce that observations from source and target domain at the same time-step have the same representations.
In our experiments, we set N to be 10% of the training trajectories.
(We also experimented with larger N ’, for example using all the training trajectories, but the differences are minor.)
In terms the alignment loss, we optimize the loss for 1000 epochs with batch size equal to 256, where at each epoch we sample a batch of paired source and target observations and compute the alignment loss.
After pre-training the target encoder, we load the weight into φ ( T ) and resumes the normal DQN or SAC training.
Our experimental results show that, although more training steps are given to the timealigned learner, it does not outperform the single-task learner, and sometimes fails to learn (e.g. in 3DBall).
The main reason is that the time-based assumption does not hold in practice as initial states are usually randomly generated.
Therefore, even though the agent exactly imitates the source-task policy at every step, the observations from source and target task do not necessarily match at every time-step.
In environments with non-deterministic transitions, the state mismatch will be a more severe issue and may lead to an unreasonable encoder.
E.1. E NVIRONMENTS Environment Settings in Vec-to-pixel Tasks
• CartPole
For the pixel-input target task, we extract the screen of the environment which is of size
(400,600), and crop the pixel input to let the image be centered at the cart.
The resulting observation has size (40,90) after cropping.
We take the difference between two consecutive frames as the agent’s observation.
• Acrobot: The source task is the same as the ordinary Acrobot environment on Gym.
For the pixel-input target task, we ﬁrst extract the screen of the environment which is of size (150,150), and then down-sample the image to (40,40).
We also take the difference between two consecutive frames as the agent’s observation.
• Cheetah-Run: The source task is the Cheetah Run Task provided by DeepMind Control Suite (DMC) (Tassa et al., 2018).
For the target task, we use the image of size (84,84) rendered from the environment as the agent’s observation.
Environment Settings in More-sensor Tasks For the target task of MuJoCo environments, we add the center of the mass based inertia and velocity into the observations of the agent, concatenating them with the original observation on the source task.
Consequently, in the target environments, the dimensionality of the observation space on target task become much larger than that of the source task.
On Hopper, the dimensionality of the target observation is 91, whereas the the source observation space only has 11 dimensions.
The dimensionalities of target tasks on HalfCheetah,
Hopper and Walker are 145, 91, 145 respectively.
Environment Settings in Broken-sensor Tasks 3DBall is an example environment provided by the ML-Agents Toolkit (Juliani et al., 2018). In this task, the agent (a cube) is supposed to balance a ball on its top. At every step, the agent will be rewarded if the ball is still on its top. If the ball falls off, the episode will immediately end.
The highest episodic return in this task is 100.
There are two versions of this game, which only differ by their observation spaces.
The simpler version (named 3DBall in the toolkit) has 8 observation features corresponding to the rotation of the agent cube, and the position and velocity of the ball.
The harder version (named 3DBallHard in the toolkit) does not have access to the ball velocity, but observes a stack of 9 past frames, each of which corresponds to the rotation of the agent cube, and the position of the ball, resulting in 45 observation dimensions at every step.
We regard 3DBall as the source task and 3DBallHard as the target task in our experiments.
E.1. I MPLEMENTATION OF B ASE DRL A LGORITHMS AND H YPER - PARAMETER S ETTINGS
Implementation of DQN To ensure that the base learning algorithm learns the pixel-input target tasks well, we follow the existing online codebases for pre-processing, architectures and hyperparameter settings in pixel CartPole 1 and pixel Acrobot 2 .
On source domain, the DQN network has a 2-layer encoder and a 2-layer Q head of hidden size 64, and the representation dimension is set as 16.
For pixel-input, the encoder has three convolution layers followed by a linear layer.
The number of channels of the convolutional layers are equal to 16, 32, 32, respectively (kernel size=5 for all three layers).
We use the Adam optimizer with learning rate 0 .
001 and β 1 , β 2 = 0 .
The target Q network is updated every 10 iterations.
In CartPole, we use a replay buffer with size 10000. In the more challenging Acrobot, we use a prioritized replay buffer with size 100000.
Implementation of SAC For MuJoCo environments, we follow an elegant open-sourced SAC implementation 3 .
The number of hidden units for all neural networks is 256.
The actor has a twolayer encoder and a two-layer policy head. The two Q networks both have three linear layers.
The activation function is ReLU and the learning rate is 3 · 10 − 4 .
We train the dynamics model and the reward model every 50k interactive steps in the source task.
For the DMC environment CheetahRun, we follow the open-sourced SAC implementation with an autoencoder 4 .
The pixel encoder has three convolution layers and one linear layer.
The number of channels for all convolutional layers is 32 and the kernel size is 3.
For the 3DBall environment, as it can only be learned within the
ML-Agents toolkit, we directly use the SAC implementation provided by the toolkit with the default hyperparameter settings.
Implementation of Latent Dynamics Model Note that our goal is to learn a good representation by enforcing it predicting the latent dynamics, different from model-based RL (Hafner et al., 2019) that aims to learn accurate models for planning. Therefore, we let the dynamics models ˆ P and ˆ R be simple linear networks, so that the representation can be more informative in terms of representing dynamics and learning values/policies. For environments with discrete action spaces, we learn |A| linear transition networks and |A| linear reward models. For environments with continuous action spaces, we ﬁrst learn an action encoder ψ : A → R d with the same encoding size d as the state representation.
Then, we learn a linear transition network and a linear reward network with ˆ
P ( φ ( o ) ◦ ψ ( a )) being the predicted next representation, and ˆ R ( φ ( o ) ◦ ψ ( a )) being the predicted reward, where ◦ denotes element-wise product.
In practice, we ﬁnd this implementation achieves good performance across many environments.
In addition, note that due to the signiﬁcant difference between source observation and target observation, the initial encoding scale could be very different in source and target tasks, making it hard for them to be regularized by the same dynamics model.
Therefore, we normalize the output of both encoders to be a unit vector (l2 norm is 1), which remedies the potential mismatch in their scales.
Hyperparameter Settings for Transfer Learning In experiments, we ﬁnd that it is better to set λ relatively large when the environment dynamics are simple and the dynamics model is of high quality.
When the environment dynamics is complex, we choose to be more conservative and set λ to be smaller.
Concretely, in CartPole, λ is set as 18; in 3DBall, λ is set as 10; in Acrobot, λ is set as 5; in the remaining MuJoCo environments where dynamics are more complicated, λ is set as 1.
Although we use different λ ’s in different environments based on domain knowledge, we ﬁnd that different values of λ ’s do not have much inﬂuence on the learning performance.
Figure 7 provided in
Appendix E.2 shows a test on the hyper-parameter λ , where we can see that our algorithm effectively transfers knowledge under various values of λ .
Regarding the representation dimension, we set it to be smaller for simpler tasks, and larger for more complex tasks.
In 3DBall, we set the encoding size to be 8; in CartPole, we set the encoding size as 16; in Acrobot, we set the encoding size as 32; in Cheetah-Run, we set the encoding size as 50; in MuJoCo tasks, we set the encoding size as 256.
Again, we ﬁnd that the feature size does not inﬂuence the performance too much.
But based on the theoretical insights of learning minimal sufﬁcient representation Achille & Soatto (2018), we believe that it is generally better to have a lower-dimensional representation while making sure it is sufﬁcient for learning.
E. A DDITIONAL E XPERIMENTAL R ESULTS Ablation Study: Transferring Different Components
Figure 5 shows the ablation study of our method in continuous control tasks.
We compare our method with the following variants:
(1) learning auxiliary tasks without transfer, (2) only transferring transition models ˆ P and (3) only transferring reward models ˆ R .
Compared with the single-task learning baseline (the blue curves)
[Figure]
we ﬁnd that all the variants of our method can make some improvements, which suggests that learning dynamics models as auxiliary tasks, transferring ˆ P and ˆ R are all effective designs for accelerating the target task learning.
Finally, our method (the red curves) that combines the above components achieves the best performance, justifying the effectiveness of our transfer algorithm [Figure]
[Figure]
Sanity Check: Effectiveness of the Proposed Transfer Method
We conduct another ablation study to evaluate each component of our algorithm in the CartPole environment as shown in Figure 6.
We ﬁnd that when transferring the dynamics models with only a linear value head (the green curve), the agent fails to learn a good policy as we analyzed in Section 4.3. If the dynamics models ( ˆ P, ˆ R ) are randomly generated instead of being transferred from the source task (the orange curve), the agent does not learn, either. More importantly, if we learn dynamics models as auxiliary tasks in the target task without transferring them from the source (the purple curve), the agent learns a little better than a vanilla agent, but is worse than our proposed transfer algorithm. These empirical results have veriﬁed our theoretical insights and shown the effectiveness of our algorithm design
[Figure]
Hyper-parameter Test
Figure 7 further visualizes how the hyperparameter λ (regularization weight) inﬂuences the transfer performance in the Vec-to-pixel CartPole environment.
It can be found that the agent generally beneﬁts from a larger λ , which suggests that the model-based regularization has a positive impact on the learning performance.
For a wide range of λ ’s, the agent always outperforms the learner without transfer (the learner with λ = 0 ).
Therefore, our algorithm is not sensitive to the hyperparameter λ , and a larger λ is preferred to get better performance.
In Appendix E.1.3, we have provided the λ selections for all experiments.