Original: A DA RL: W HAT , W HERE , AND H OW TO A DAPT IN T RANSFER R EINFORCEMENT L EARNING

Original: A BSTRACT

Original: One practical challenge in reinforcement learning (RL) is how to make quick adaptations when faced with new environments.

Original: In this paper, we propose a principled framework for adaptive RL, called AdaRL , that adapts reliably and efﬁciently to changes across domains with a few samples from the target domain, even in partially observable environments.

Original: Speciﬁcally, we leverage a parsimonious graphical representation that characterizes structural relationships over variables in the RL system.

Original: Such graphical representations provide a compact way to encode what and where the changes across domains are, and furthermore inform us with a minimal set of changes that one has to consider for the purpose of policy adaptation.

Original: We show that by explicitly leveraging this compact representation to encode changes, we can efﬁciently adapt the policy to the target domain, in which only a few samples are needed and further policy optimization is avoided.

Original: We illustrate the efﬁcacy of AdaRL through a series of experiments that vary factors in the observation, transition and reward functions for Cartpole and Atari games.

Original: I NTRODUCTION AND R ELATED W ORK

Original: Over the last decades, reinforcement learning (RL) (Sutton and Barto, 1998) has been successful in many tasks (Mnih et al., 2013; Silver et al., 2016).

Original: Most of these early successes focus on a ﬁxed task in a ﬁxed environment.

Original: However, in real applications we often have changing environments, and it has been demonstrated that the optimal policy learned in a speciﬁc domain may not be generalized to other domains (Taylor and Stone, 2009).

Original: In contrast, humans are usually good at transferring acquired knowledge to new environments and tasks both efﬁciently and effectively (Pearl and Mackenzie, 2018), thanks to the ability to understand the environments.

Original: Generally speaking, to achieve reliable, low-cost, and interpretable transfer, it is essential to understand the underlying process—which decision-making factors have changes, where the changes are, and how they change, instead of transferring blindly (e.g., transferring the distribution of high-dimensional images directly).

Original: There are roughly two research lines in transfer RL (Taylor and Stone, 2009; Zhu et al., 2020): (1) ﬁnding policies that are robust to environment variations, and (2) adapting policies from the source domain to the target domain as efﬁciently as possible.

Original: For the ﬁrst line, the focus is on learning policies that are robust to environment variations, e.g., by maximizing a risk-sensitive objective over a distribution of environments (Tamar et al., 2015) or by extracting a set of invariant states (Zhang et al., 2020a; 2021a; Tomar et al., 2021).

Original: A more recent method encodes task-relevant invariances by putting behaviorally equivalent states together, which helps better generalization (Agarwal et al., 2021).

Original: On the other hand, with the increase of the number of domains, the common part may get even smaller, running counter to the intention of collecting more information with more domains.

Original: Moreover, focusing only on the invariant part and disregarding domain-speciﬁc information may not be optimal; for instance, in the context of domain adaptation, it has demonstrated that the variable part also contains information helpful to improve prediction accuracy (Zhang et al., 2020b).

Original: In this paper, we propose a method along the second line, adapting source policies to the target.

Original: Approaches along this line adapt knowledge from source domains and reuse it in the target domain to improve data efﬁciency, i.e., in order for the agent to require fewer explorations to learn the target policy.

Original: For example, an agent could use importance reweighting on samples (cid:104) s, a, r, s (cid:48) (cid:105) from sources (Tirinzoni et al., 2018; 2019) or start from the optimal source policy to initialize a learner in the target domain, as a near-optimal initializer (Taylor et al., 2007; Fernández et al., 2010).

Original: Another widely-used technique is ﬁnetuning: a model is pretrained on a source domain and the output layers are ﬁnetuned via backpropagation in the target domain (Hinton and Salakhutdinov, 2006; Mesnil et al., 2012).

Original: PNNs (Rusu et al., 2016), instead, retain a pool of pretrained models and learn lateral connections from them to extract useful features for a new task.

Original: Moreover, a set of approaches focus on sim2real transfer by adapting the parameters (Yu et al., 2017; Peng et al., 2020).

Original: However, many of these approaches still require a large amount of explorations and optimization in the target domain.

Original: Recently, meta-RL approches such as MAML (Finn et al., 2017), PEARL (Rakelly et al., 2019),

Original: CAVIA (Zintgraf et al., 2019), Meta-Q learning (Fakoor et al., 2020), and others (Mendonca et al., 2019; Nagabandi et al., 2018; Duan et al., 2016) have been successfully applied to learn an inductive bias that accelerates the learning of a new task by training on a large number of tasks.

Original: Some of these methods (e.g., CAVIA and PEARL), as well as some prior work (e.g., HiMDPs (Doshi-Velez and

Original: Konidaris, 2016)) and recent follow-ups (Zhang et al., 2021b), have a similar motivation to our work: in a new environment not all parameters need to be updated, so we can force the model to only adapt a set of context parameters .

Original: However, these methods mostly focus on MDPs (except the Block MDP assumption in Zhang et al.

Original: (2021b))

Original: and model all changes as a black-box, which may be less efﬁcient for adaptation, as opposed to a factorized representation of change factors.

Original: Considering these limitations, we propose AdaRL, a transfer RL approach that achieves low-cost, reliable, and interpretable transfer for partially observable Markov decision processes (POMDPs), with MDPs as a special case.

Original: In contrast to state-of-the-art approaches, we learn a parsimonious graphical representation that is able to characterize structural relationships among different dimensions of states, change factors, the perception, the reward variable, and the action variable.

Original: It allows us to model changes in transition, observation, and reward functions in a component-wise way.

Original: This representation is related to Factored MDPs (Kearns and Koller, 1999; Boutilier et al., 2000; Strehl et al., 2007) and Factored POMDPs (Katt et al., 2019), but augmented with change factors that represent a low-dimensional embedding of the changes across domains.

Original: Our main motivation is that distribution shifts are usually localized – they are often due to the changes of only a few variables in the generative processes, so we can just adapt the distribution of a small portion of variables (Huang et al., 2020; Schölkopf et al., 2021) and, furthermore, factorized according to the graph structure distribution module

Original: In Fig.

Original: 1 we give a motivating example and a general description of AdaRL.

Original: In this example, we consider learning policies for Pong (Bellemare et al., 2013) that can easily generalize to different rotations ω and to images corrupted with white noise.

Original: Speciﬁcally, given data from n source domains with different rotations and noise variances, we learn a parsimonious latent state representation shared by all domains, denoted by s t , and characterize the changes across domains by a two-dimensional factor θ k .

Original: We identify a set of minimal sufﬁcient representations ( s mint , θ mink ) for policy transfer.

Original: For instance, here only the rotation factor ω needs adapting (i.e., θ mink = ω k ), since the noise factor does not affect the optimal policy.

Original: Similarly, as we will show formally in the rest of the paper, not all components s i,t of the state vector s t are necessary for policy transfer.

Original: For example, s 2 ,t (cid:54)∈ s mint , since it never affects the future reward.

Original: We learn an optimal policy π ∗ ( ·| θ mink ) on source domains.

Original: In the target domain, we only need a few samples to quickly estimate the value of the low-dimensional θ min target , and then we can apply π ∗ ( ·| θ mintarget ) directly.

Original: Our main contributions are summarized below:

Original: • We assume a generative environment model, which explicitly takes into account the structural relationships among variables in the RL system.

Original: Such graphical representations provide a compact way to encode what and where the changes across domains are.

Original: • Based on this model, we characterize a minimal set of representations that sufﬁce for policy learning across domains, including the domain-speciﬁc change factors and domain-shared state representations.

Original: With this characterization, we adapt the policy with only a few target samples and without policy optimization in the target domain, achieving low-cost and reliable policy transfer.

Original: • By leveraging a compact way to encode the changes, we also beneﬁt from multi-task learning in model estimation.

Original: In particular, we propose the Multi-model Structured Sequential Variational Auto-Encoder (MiSS-VAE) for reliable model estimation in general cases.

Original: A C OMPACT R EPRESENTATION OF E NVIRONMENTAL S HIFTS

Original: Suppose there are n source domains and n (cid:48) target domains.

Original: In each source domain, we observe sequences {(cid:104)

Original: o t , a t , r t (cid:105)}

Original: Tt =1 , where o t ∈ O are the perceived signals at time t

Original: (e.g., images), a t ∈ A is the executed action, and r t ∈ R is the reward signal.

Original: We denote the underlying latent states by s t =

Original: ( s 1 ,t , · · · , s d,t ) (cid:62) , where d is the dimensionality of latent states.

Original: We assume that the generative process of the environment in the k -th domain (with k = 1 , . . .

Original: n + n (cid:48) ) can be described in terms of the transition function for each dimension of s and the observation and reward functions as

Original: [Equation]

Original: (1) where (cid:12) denotes the element-wise product, the (cid:15) si,t , (cid:15) ot , (cid:15) rt terms are i.i.d.

Original: random noises.

Original: As explained below, c · (cid:41) · are masks (binary vectors or scalars that represent structural relationships from one variable to the other), and θ k

Original: = ( θ s k , θ ok , θ rk ) are the change factors that have a constant value in each domain, but vary across domains in the transition, observation, and reward function, respectively.

Original: The latent states s t +1 form an MDP: given s t and a t , s t +1 is independent of previous states and actions.

Original: The perceived signals o t are generated from the underlying states s t .

Original: The actions a t directly inﬂuence the latent states s t +1 , instead of the observed signals o t , and the reward is determined by the latent states and the action.

Original: Eq.

Original: 1 can also represent MDPs as a special case if states s t are directly observed, in which case the observation function of o t is not needed.

Original: Structural relationships and graphs.

Original: Often the action variable a t − 1 does not inﬂuence every dimension of s t , and similarly, the reward r t may not be inﬂuenced by every dimension of s t − 1 .

Original: Furthermore, there are structural relationships between different dimensions of s t − 1 and s t .

Original: To characterize these constraints, we explicitly take into account the graph structure G over the variables in the system characterized by a Dynamic Bayesian Network (Murphy, 2002) and encode the edges with masks c · (cid:41) · .

Original: In the ﬁrst equation in Eq.

Original: 1 the transition function for the state component s i , where the j th entry of c s (cid:41) s i ∈ { 0 , 1 } d is 1 if

Original: [-and only if s j,t inﬂuences s i,t +1 (graphically represented by an edge), while c a (cid:41) s i ∈ { 0 , 1 } is 
-]
Revised: [-and only if s j,t inﬂuences s i,t +1 (graphically represented by an edge), while c a (cid:41) s i ∈ { 0 , 1 } is 
-]

Original: if and only if the action a t has any effect on s i,t +1 .

Original: Similarly, the binary vector c θ k (cid:41) s i ∈ { 0 , 1 } p encodes which components of the change factor θ s k

Original: = ( θ s 1 ,k , . .

Original: . , θ sp,k ) (cid:62) affect s i,t +1 .

Original: The masks in the observation function g and reward function h have similar functions.

Original: The masks and the parameters of the functions f , g , and h , are invariant; all changes are encoded in θ k .

Original: For simplicity of notation, we collect all the transition mask vectors in the matrices C s (cid:41) s :=

Original: [ c s (cid:41) s i ] di = and C θ k (cid:41) s :=

Original: [ c θ k (cid:41) s i ]

Original: d i =1 and the scalars in the vector c a (cid:41) s :=

Original: [ c a (cid:41) s i ]

Original: d i =1 .

Original: Characterization of change factors in a compact way.

Original: In practical scenarios, the environment model may change across domains.

Original: Moreover, it is often the case that given a high-dimensional input, only a few factors may change, which is known as minimal change principle (Ghassami et al., 2018) or sparse mechanism shift assumption (Schölkopf et al., 2021).

Original: In such a case, instead of learning the distribution shift over the high-dimensional input, thanks to the parsimonious graphical representation, we introduce a low-dimensional vector θ k to characterize the domain-speciﬁc information in a compact way (Zhang et al., 2020b).

Original: Speciﬁcally, θ ok , θ rk , and θ s k capture the change factors in the observation function, reward function, and transition dynamics, respectively; each of them can be multi-dimensional and that they are constant within each domain.

Original: In general, θ k can capture both the changes in the inﬂuencing strength and those in the graph structure, e.g., some edges may appear only in some domains.

Original: Since we assume that the structural relationships in Eq.

Original: 1 are invariant across domains, this means that the masks c · (cid:41) · have to encode an edge even if it presents only in one domain, and furthermore, since θ k encodes the changes, it can switch the edge off in other domains.

Original: Fig.shows an example of the graphical representation of the (estimated) environment model.

Original: Speciﬁcally, in this example, θ sk only inﬂuences s 1 ,t , a t − 1 does not have an edge to s 1 ,t , and among the states, only s d,t − 1 has an edge to r t .

Original: In this example, we consider the case when the control signals are random, so there is no edge between s t and a t .

Original: We ﬁrst assume that the environment model in Eq. 1 is known (we will explain how to learn it in Sec. 3.1), and characterize which changes have an effect on the policy transfer to the target domain.
Revised: A PPENDIX FOR “A DA RL: W HAT , W HERE , AND H OW TO A DAPT IN T RANSFER R EINFORCEMENT L EARNING "

Original: In Eq.

Original: 1, we allow the model to change across domains, including all involved functions, and we leverage θ k to capture the changes in a compact way.

Original: The varying model implies that the optimal policy function may also vary across domains.

Original: How can we then characterize the changes in the optimal policy function in a compact way, as we did in the model?

Original: Interestingly, we ﬁnd that the change factor θ k and the latent state s t are sufﬁcient for policy learning, but not every dimension of θ k or s t is necessary, since they may not ever have an effect on the reward, even in future steps.

Original: We ﬁrst give the deﬁnitions of compact domain-shared representations and compact domain-speciﬁc representations , according to the graph structure, and we further show that they are the minimal set of dimensions that sufﬁce for policy learning across domains (proof in Appendix).

Original: Deﬁnition 1.

Original: Given the graphical representation of an environment model G that is encoded in the binary masks c · (cid:41) · , we deﬁne recursively the representations that affect the reward in the future as:

Original: • compact domain-shared representations s mint : the latent state components s i,t ∈ s t that either – have an edge to the reward in the next time-step r t +1 , i.e., c s (cid:41) r i = 1 , or – have an edge to another state component in the next time-step s j,t +1 , i.e., c s (cid:41) s j,i = 1 , such that the same component at time t is a compact domain-shared representation, i.e., s j,t ∈ s mint ; • compact domain-speciﬁc representations θ mink : the latent change factors θ

Original: i,k ∈ θ k that either: – have an edge to the reward in the next time-step r t +1 , i.e., θ i,k

Original: = θ rk

Original: and c θ k (cid:41) r = 1 , or – have an edge to a state component s j,t ∈ s min t , i.e., c θ k (cid:41) s j,

Original: i = 1 .

Original: Proposition 1.

Original: Under the assumption that the graph G is Markov and faithful to the measured data, the union of compact domain-speciﬁc θ mink and compact shared representations s mint are the minimal and sufﬁcient dimensions for policy learning across domains.

Original: For the example in Fig.

Original: 1, s mint =

Original: ( s 1 ,t , s d,t ) and θ mink = { θ sk , θ rk } .

Original: Note that θ ok is never in θ mink , and thus if only the observation function changes, the optimal policy function π ∗ k remains the same across domains.

Original: For example in Cartpole a change of color does not affect the optimal policy.

Original: Moreover, if c θ k (cid:41) r = 1 , then θ rk ∈ θ mink , which is the case for multi-task learning.

Original: 3. S IMULTANEOUS E STIMATION OF D OMAIN -V ARYING M ODELS

Original: In this section, we give the estimation procedure of the environment model in Eq.

Original: 1 from observed sequences {{(cid:104)

Original: y t,k , a t,k (cid:105)}

Original: Tt =1 } nk =1 from each source domain k , where y t,k

Original: =

Original: ( o (cid:62) t,k , r (cid:62) t,k ) (cid:62) are the observations and reward at time t in domain k .

Original: Instead of estimating the model in each domain separately, we estimate models from different domains simultaneously, by exploiting commonalities across domains while at the same time preserving speciﬁc information for each domain.

Original: In particular, we propose the Multi-model Structured Sequential Variational Auto-Encoder (MiSS-VAE) , which contains the following three essential components.

Original: (1) "Sequential VAE" component handles the sequential data, with the underlying latent states satisfying an MDP.

Original: It is implemented by adding an

Original: LSTM (Hochreiter and Schmidhuber, 1997) to encode the sequential information in the encoder to learn the inference model q φ ( s t,

Original: k | s t − 1 ,k , y 1: t,k , a 1: t − 1 ,k ; θ k ) .

Original: (2) "Multi-model" component handles models from different domains at the same time, using the domain index k as an input and learning the domain-speciﬁc factors θ k .

Original: (3) "Structured" component : exploits the structural information that is explicitly encoded with the binary masks, i.e., c · (cid:41) · in Eq.

Original: Here, the joint distribution of latent states are factorized according to structures, instead of being marginally independent as in traditional VAEs (Kingma and Welling, 2013).

Original: Fig.

Original: A3 in Appendix gives the diagram of neural network architecture in model training. Let y 1: n 1: T = {{ y t,k } Tt =1 } nk =1 . By taking into account the above three components, we maximize the following objective function L : [Equation]
Revised: Appendix organization:

Original: In particular, L rec is the reconstruction loss for both observed images and rewards, to learn the observation and the reward function, respectively.

Original: We also consider the one-step prediction loss L pred .

Original: [Equation] L pred = n (cid:80) k = T − 2 (cid:80) t =
Revised: • Appendix A1:

Original: [Equation]

Original: where p β 1 and p β 2 denote the generative models with parameters β 1 and β 2 , respectively, that are shared across domains, and q φ the inference model with shared parameters φ .

Original: We also use the following KL-divergence loss to constrain the latent space:

Original: [Equation]

Original: where we explicitly model the transition dynamics p γ with the parameters γ shared across domains; this is essential for establishing a Markov chain in latent space and learning a representation for long-term predictions.

Original: Moreover, the KL loss helps to constrain the latent space to (1) ensure that the disentanglement between the inferred latent factors q ( s i,t |· ) and q ( s j,t |· ) for i (cid:54) =

Original: j , since we do not consider the instantaneous connections among state dimensions, and (2) ensure that the latent representations s t are maximally compressive about the observed high-dimensional data.

Original: Furthermore, according to the edge-minimality property (Zhang and Spirtes, 2011) and the minimal change principle (Ghassami et al., 2018), we add sparsity constraints on structural matrices and on the change of domain-speciﬁc factors across domains, respectively, to achieve better identiﬁability:

Original: [Equation] [Table] [Equation] Note that besides the shared parameters { β 1 , β 2 , φ, γ } , the structural relationships (encoded in binary masks c ) are also involved in the shared parameters.
Revised: Proof of Proposition• Appendix A2: Proof of Theorem• Appendix A3:

Original: Each factor in p φ , p β i , and p γ is modeled with a mixture of Gaussians, because with a suitable number of Gaussians, it can approximate a wide class of continuous distributions.

Original: Moreover, in model estimation, the domain-speciﬁc factors θ k = { θ ok , θ sk , θ rk } are treated as parameters; they are constant within the same domain, but may differ in different domains.

Original: We explicitly consider θ k not only in the generative models p β i and p γ , but also in the inference model q φ .

Original: In this way, except for θ k , all other parameters in MiSS-VAE are shared across domains, so that all we need to update in the target domain is the low-dimensional θ k , which greatly improves the sample efﬁciency and the statistical efﬁciency in the target domain.

Original: 3. L OW -C OST AND I NTERPRETABLE P OLICY T RANSFER

Original: After identifying what and where to transfer, we show how to adapt.

Original: Instead of learning the optimal policy in each domain separately, which is time and sample inefﬁcient, we leverage a multi-task learning strategy: policies in different domains are optimized at the same time exploiting both commonalities and differences across domains.

Original: Given the compact domain-shared s mint and domainspeciﬁc representations θ mink , we represent the optimal policies across domains in a uniﬁed way:

Original: [Equation]

Original: where θ mink explicitly and compactly encodes the changes in the policy function in each domain k , and all other parameters in the optimal policy function π ∗ are shared across domains.

Original: In other words, by learning π ∗ in the source domains, and estimating the value of the change factor θ min target and inferring latent states s min target from the target domain, we can immediately derive the optimal policy in the target domain without further policy optimization by just applying Eq. 2.

Original: The AdaRL framework answers what and where the change factors are and which change factors need to adapt across domains in an interpretable way.

Original: Moreover, AdaRL only requires a few samples to update the low-dimensional domain-speciﬁc parameters θ min target to achieve the optimal policy in the target domain, without further policy optimization, achieving the low cost.

Original: We provide the pseudocode for the AdaRL algorithm in Alg. 1.

Original: The algorithm has three parts: (1) data collection with a random policy or any initial policy from n source domains (line 2), (2) model estimation from the n source domains with multi-task learning (lines 2-3, see Sec.

Original: 3.1 for details), and (3) learning the optimal policy π ∗ with deep Q-learning, by making use of domain-speciﬁc factors and the inferred domain-shared state representations (lines 4-21).

Original: Speciﬁcally, because we do not directly observe the states s t , we infer q ( s mint +1 ,k | o ≤ t +1 ,k , r ≤ t +1 ,k , a ≤ t,k , θ mink ) and sample s mint +1 ,k from its posterior, for the k th domain (lines 7 and 13).

Original: Moreover, the action-value function Q is learned by considering the averaged error over the n source domains (line 18).

Original: AdaRL can be implemented with a wide class of policy-learning algorithms, e.g., DDPG (Lillicrap et al., 2015), Q-learning (Mnih et al., 2015), and

Original: Actor-Critic methods (Schulman et al., 2015; Mnih et al., 2016).

Original: Then, in the target domain, we only need to collect a few rollouts to estimate the low-dimensional domain-speciﬁc representations θ min target , with all other parameters being ﬁxed (lines 22-23).

Original: Algorithm 1 (AdaRL with Domains Shifts)

Original: 3: Identify the dimension indices of s mint and the values of θ mink according to the learned model.

Original: 4: for episode = 1, ..., M do 5: for source domain k = 1, ..., n do 6:

Original: Receive initial observations o 1 ,k and r 1 ,k for the k -th domain.

Original: Infer the posterior q ( s min 1 ,k | o 1 ,k , r 1 ,k , θ mink ) and sample initial inferred state s min 1 ,k .

Original: 8: end for 9: for timestep t = 1, ..., T do 10: for source domain k = 1, ..., n do 11:

Original: Select a t,k randomly with probability (cid:15) ; otherwise a t,k = arg max a Q ( s mint,k , a, θ mink ) .

Original: Execute action a t,k , and receive reward r t +1 ,k and observation o t +1 ,k in the k th domain.

Original: Infer the posterior q ( s mint +1 ,k | o ≤ t +1 ,k , r ≤ t +1 ,k , a ≤ t,k , θ mink ) and sample s mint +1 ,k .

Original: Store transition ( s mint,k , a t,k , r t +1 ,k , s mint +1 ,k , θ mink ) in reply buffer B .

Original: 15: end for 16:

Original: Randomly sample a minibatch of N transitions ( s mini,j , a i,j , r i +1 ,j , s mini +1 ,j , θ minj ) from B .

Original: Set y i,j = r i +1 ,j + λ max a (cid:48) Q (cid:48)

Original: ( s mini +1 ,j , a (cid:48) , θ minj ) .

Original: Update action-value function Q by minimizing the loss:

Original: L = n ∗ N (cid:88) i,j ( y i,j − Q ( s mini,j , a i,j , θ minj )) 2 .

Original: 19: end for 20:

Original: Update the target network Q (cid:48) : Q (cid:48)

Original: =

Original: Q .

Original: 21: end for 22: Record a few rollouts from the target domain.

Original: 23: Estimate the values of θ min target for the target domain, with all other parameters ﬁxed.

Original: 3. T HEORETICAL P ROPERTIES

Original: Below we show the conditions under which we can identify the true graph G from observational data, even when the model in Eq. 1 is unknown. Furthermore, we derive a generalization bound of the state-value function under the PAC-Bayes framework (McAllester, 1999). Theorem 1 (Structural Identiﬁability) .
Revised: Proof of Theorem• Appendix A4: More details for model estimation • Appendix A5: Complete experimental results • Appendix A6:

Original: Suppose the underlying states s t are observed, i.e., Eq.

Original: (1) is an MDP.

Original: Then under the Markov condition and faithfulness assumption, the structural matrices

Original: C s (cid:41) s , c a (cid:41) s , c s (cid:41) r , c a (cid:41) r , C θ k (cid:41) s , and c θ k (cid:41) r are identiﬁable.

Original: This theorem shows that in the MDP scenario, where the underlying states are observed and c θ k (cid:41) o and c s (cid:41) o are not considered by deﬁnition, we can uniquely determine the structural relationships over { s t − 1 , s t , a t − 1 , r t , θ k } , i.e., the Dynamic Bayesian network G , from observed data under mild

Original: [Figure]

Original: Hard

Original: [Figure]

Original: coded

Original: Reward Racket

Original: Pong d

Original: L

Original: Reward conditions, without knowing the generative environment model.

Original: Even if θ k is not directly observed, we can identify which state dimension changes and if there is a change in the reward function.

Original: Suppose there are n source domains, and for the k th domain, we have S k

Original: = (cid:0) ( s 1 ,k , v ∗ ( s 1 ,k )) , · · · , ( s m k ,k , v ∗ ( s m k ,k )) (cid:1) , where m k is the number of samples from the k th domain, s · ,k is a state sampled from the k th domain, and v ∗ ( s · ,k ) is its corresponding optimal state-value.

Original: For any value function h θ mink ( · ) parameterized by θ mink , we deﬁne the loss function (cid:96) ( h θ mink , ( s k,i , v ∗ ( s i,k )))

Original: =

Original: D dist ( h θ mink ( s i,k ) , v ∗ ( s i,k )) , where D dist is a distance function that measures the discrepancy between the learned value and the optimal value.

Original: The following theorem gives a generalization bound of the state-value function under the PAC-Bayes framework.

Original: Theorem 2 (Generalization Bound) .

Original: Let Q be an arbitrary distribution over θ mink and P the prior distribution over θ mink .

Original: Then for any δ ∈ (0 , 1] , with probability at least 1 − δ , the following inequality holds uniformly for all Q ,

Original: [Equation]

Original: where er ( Q ) and ˆ er ( Q , S k ) are the generalization error and the training error between the estimated value and the optimal value, respectively.

Original: Theorem 2 states that with high probability the generalization error er ( Q ) is upper bounded by the empirical error plus two complexity terms.

Original: Speciﬁcally, the ﬁrst one is the average of the taskcomplexity terms from the observed domains, which converges to zero in the limit of samples in each domain, i.e., m k → ∞ .

Original: The second is an environment-complexity term, which converges to zero if inﬁnite domains are observed, i.e., n → ∞ .

Original: Moreover, if assuming different dimensions of θ mink are independent, then D KL ( Q||P ) =

Original: (cid:80) | θ mink | i =

Original: D KL ( Q i ||P i ) , which indicates that a low-dimensional θ mink usually has a smaller KL divergence, so does the upper bound of the generalization error.

Original: E VALUATION

Original: We modify the Cartpole and Atari Pong environments in OpenAI Gym (Brockman et al., 2016).

Original: Here, we present a subset of the results; see Appendix for the complete results and the detailed settings.

Original: We consider changes in the state dynamics (e.g., the change of gravity or cart mass in Cartpole, change of orientation in Pong), changes in observations (e.g., different noise levels in images or different colors in Pong), and changes in reward functions (e.g., different reward functions in Pong based on the contact point of the ball), as shown in Fig.

Original: 2 for Pong.

Original: For each of these factors, we take into account both interpolation (where the factor value in the target domain is in the support of that in source domains), and extrapolation (where it is out of the support w.r.t.

Original: the source domains).

Original: We train on n source domains based on the trajectory data generated by a random policy.

Original: In Cartpole, for each domain we collect 10000 trials with 40 steps.

Original: For Pong experiments, each domain contains data each of them takes a maximum of 10000 steps.

Original: In the target domain we consider different sample sizes with N target = { 20 , 50 , 10000 } to estimate θ min target .

Original: For both games, we evaluate the POMDP case, where the inputs are high-dimensional images; note that we did not stack multiple frames, so some properties (e.g., velocity) are not observed, resulting in a POMDP.

Original: For Cartpole, we also consider the MDP case, where the true states (cart position and velocity, pole angle and angular velocity) are used as the input to the model.

Original: In Cartpole, we also experiment with multiple factors changing at the same time (e.g., gravity and mass change concurrently in the target domain).

Original: [Table]

Original: Modiﬁed Cartpole setting The Cartpole problem consists of a cart and a vertical pendulum attached to the cart using a passive pivot joint.

Original: The task is to prevent the vertical pendulum from falling by putting a force on the cart to move it left or right.

Original: We introduce two change factors for the state dynamics θ sk : varying gravity and varying mass of the cart.

Original: In terms of changes on the observation function θ ok , we add Gaussian noise on the images.

Original: Since θ ok does not inﬂuence the optimal policy (as shown in Prop. 1), we need it only for the model estimation, but not for policy optimization.

Original: Moreover, if θ k = { θ ok } , the optimal policy is shared across domains.

Original: Modiﬁed Pong setting In Pong, one of the established Atari benchmarks (Bellemare et al., 2013), the agent controls a paddle moving up and down vertically, aiming at hitting the ball.

Original: We consider changes in observation function θ ok , state dynamics θ sk , and reward function θ rk , as shown in Fig.

Original: We consider three change factors on perceived signals θ ok : different image sizes, different image colors, and different noise levels.

Original: For the setting with different image colors, we use RGB images as inputs and consider source domains with varying RGB colors {original, green, red} and target domains with colors {yellow, white}, but for other settings, we convert the images to grayscale as input.

Original: To change the state dynamics, we rotate the images ω degrees clockwise.

Original: To test the changes in the reward function, we model the reward as a function of the distance between contact point and the central point of the paddle, denoted by d , as opposed to the original Pong in which it is constant.

Original: We denote by L the half-length of the paddle and formulate two groups of reward functions: (1) Linear reward: r t = αdL ; and (2) Non-linear reward: r t =

Original: αLd +3 L , where α varies across domains.

Original: Baselines In the MDP setting, we compare AdaRL with CAVIA (Zintgraf et al., 2019) and

Original: PEARL (Rakelly et al., 2019).

Original: In the POMDP setting, we compare with PNN (Rusu et al., 2016),

Original: PSM (Agarwal et al., 2021) and MQL (Fakoor et al., 2020).

Original: We also compare with AdaRL* , a version of AdaRL that does not learn the binary masks c · (cid:41) · and therefore does not use any structural information.

Original: All of these methods use the same number of samples N target from the target domain.

Original: We also compare with: 1) Non-t , a vanilla non-transfer baseline that pools data from all source domains and learns a ﬁxed model; and 2) an oracle baseline, which is completely trained on the target domain with model-free exploration.

Original: For a fair comparison, we use the same policy learning algorithm, Double DQN (Van Hasselt et al., 2016), for all methods.

Original: As opposed to MAML and PNN,

Original: AdaRL only uses the N target samples to estimate θ mintarget , without any policy optimization.

Original: Results We measure performance by the mean and standard deviation of the ﬁnal scores overtrials with different random seeds, as well as testing the signiﬁcance with the Wilcoxon signedrank test (Conover, 1999).

Original: As shown in Tables 1, 2 and 3 1 , AdaRL consistently outperforms the baselines across most change factors in the MDP and POMDP case for modiﬁed Cartpole, and in the POMDP case for Pong for N target = 50 .

Original: As ablation studies, to see the effect of learning In Table 1-3, [Table] [Table]
Revised: Experimental details

Original: the graphical structure, we also compare with AdaRL ∗ , which does not learn the binary masks c · , but just assumes everything is fully connected.

Original: Learning the graphical structure improves the performances signiﬁcantly, and without it AdaRL ∗ is generally comparable to baselines.

Original: We provide results with N target = { 20 , 50 , 10000 } in Appendix, showing that the performance gains are larger at smaller sample sizes.

Original: Furthermore, we consider the change of reward functions (see Table A12in Appendix).

Original: More detailed experimental results are provided in Appendix, including the average score across different N target , policy learning curves and an analysis of the estimated θ k w.r.t.

Original: real change factor.

Original: Interestingly, in the Cartpole case, the estimated θ k matches the physical quantities that are being changed across the domains.

Original: In particular, the estimated θ k for gravity and noise are linear mappings of the gravity and noise level values.

Original: For the mass-varying case, the learned θ sk is a nonlinear monotonic function of the mass, which matches the inﬂuence of the mass on the dynamics.

Original: C ONCLUSIONS AND FUTURE WORK

Original: In this paper, we proposed AdaRL, a principled framework for transfer RL.

Original: AdaRL learns a latent representation with domain-shared and domain-speciﬁc components across source domains, uses it to learn an optimal policy parameterized by the domain-speciﬁc parameters, and applies it to a new target domain.

Original: It is achieved without any further policy optimization, but just by estimating the values of the domain-speciﬁc parameters in the target domain, which can be accomplished with a few target-domain data.

Original: As opposed to previous work, AdaRL can model changes in the state dynamics, observation function and reward function in an uniﬁed manner, and exploit the factorization to improve the data efﬁciency and adapt faster with fewer samples.

Original: Further directions include exploiting the target domain to ﬁne-tune the policy and handling the out-of-distribution data.

Original: Moreover, exploring an alternative to the reconstruction loss, e.g., using the contrastive loss (Srinivas et al., 2020), might also improve the training efﬁciency.

Original: Finally, an exciting next step is to transfer knowledge across different tasks, e.g., different Atari games.

Original: R EPRODUCIBILITY S TATEMENT

Original: Proofs of all our theoretical results are given in Appendix A1, A and A3 with disclosure of all assumptions.

Original: More details about the model estimation are given in Appendix A4.

Original: Appendix A provides the complete experimental details and results.

Original: All datasets used are publicly available or instructions are provided on how to generate them in Appendix 5 and 6.

Original: A description of the hyperparameters and network architectures used is included in Appendix 6.

Original: Source code for all our experiments are attached in the supplemental material and will be made available on GitHub after deanonymisation, providing also a complete description of our experimental environment, conﬁguration ﬁles and instructions on the reproduction of our experiments.

Original: A P ROOF OF P ROPOSITION
Revised: A P ROOF OF P ROPOSITION

Original: We ﬁrst review the deﬁnitions of d-separation, the Markov condition, and the faithfulness assumption (Spirtes et al., 1993; Pearl, 2000), which will be used in the proof.
Revised: We ﬁrst review the deﬁnitions of d-separation, the Markov condition, and the faithfulness assumption (Spirtes et al., 1993; Pearl, 2000), which will be used in the proof.

Original: Given a directed acyclic graph G =
Revised: Given a directed acyclic graph G =

Original: ( V , E ) , where V is the set of nodes and E is the set of directed edges, we can deﬁne a graphical criterion that expresses a set of conditions on the paths.
Revised: ( V , E ) , where V is the set of nodes and E is the set of directed edges, we can deﬁne a graphical criterion that expresses a set of conditions on the paths.

Original: Deﬁnition A1 (d-separation
Revised: Deﬁnition A1 (d-separation

Original: Pearl, 2000)) .
Revised: Pearl, 2000)) .

Original: A path p is said to be d-separated by a set of nodes
Revised: A path p is said to be d-separated by a set of nodes

Original: Z ⊆ V if and only if (1) p contains a chain i
Revised: Z ⊆ V if and only if (1) p contains a chain i

Original: → m → j or a fork i ← m → j such that the middle node m is in Z , or (2) p contains a collider i → m ← j such that the middle node m is not in Z and such that no descendant of m is in Z .
Revised: → m → j or a fork i ← m → j such that the middle node m is in Z , or (2) p contains a collider i → m ← j such that the middle node m is not in Z and such that no descendant of m is in Z .

Original: Let X , Y , and Z be disjunct sets of nodes.
Revised: Let X , Y , and Z be disjunct sets of nodes.

Original: Z is said to d-separate X from Y (denoted as X ⊥ d Y | Z ) if and only if Z blocks every path from a node in X to a node in Y .
Revised: Z is said to d-separate X from Y (denoted as X ⊥ d Y | Z ) if and only if Z blocks every path from a node in X to a node in Y .

Original: Deﬁnition A2 (Global Markov Condition (Spirtes et al., 1993; Pearl, 2000)) .
Revised: Deﬁnition A2 (Global Markov Condition (Spirtes et al., 1993; Pearl, 2000)) .

Original: A distribution P over
Revised: A distribution P over

Original: V satisﬁes the global Markov condition on graph G if for any partition ( X, Z, Y ) such that X ⊥ d Y | Z
Revised: V satisﬁes the global Markov condition on graph G if for any partition ( X, Z, Y ) such that X ⊥ d Y | Z

Original: [Equation]
Revised: [Equation]

Original: In other words, X is conditionally independent of Y given Z , which we denote as X ⊥⊥ Y | Z .
Revised: In other words, X is conditionally independent of Y given Z , which we denote as X ⊥⊥ Y | Z .

Original: Deﬁnition A3 (Faithfulness Assumption (Spirtes et al., 1993; Pearl, 2000)) .
Revised: Deﬁnition A3 (Faithfulness Assumption (Spirtes et al., 1993; Pearl, 2000)) .

Original: There are no independencies between variables that are not entailed by the Markov Condition.
Revised: There are no independencies between variables that are not entailed by the Markov Condition.

Original: If we assume both of these assumptions, then we can use d-separation as a criterion to read all of the conditional independences from a given DAG G .
Revised: If we assume both of these assumptions, then we can use d-separation as a criterion to read all of the conditional independences from a given DAG G .

Original: In particular, for any disjoint subset of nodes
Revised: In particular, for any disjoint subset of nodes

Original: X,Y,Z ⊆ V : X ⊥⊥ Y
Revised: X,Y,Z ⊆ V : X ⊥⊥ Y

Original: [Equation]
Revised: [Equation]

Original: In our case we can represent the generative model in Eq.
Revised: In our case we can represent the generative model in Eq.

Original: 1 as a Dynamic Bayesian Network(DBN)
Revised: 1 as a Dynamic Bayesian Network(DBN)

Original: G (Murphy, 2002) over the variables { s t − 1 , a t − 1 , o t − 1 , r t , s t , θ k } , where the binary masks c · (cid:41) · represent edges or sets of edges, as shown in Fig.
Revised: G (Murphy, 2002) over the variables { s t − 1 , a t − 1 , o t − 1 , r t , s t , θ k } , where the binary masks c · (cid:41) · represent edges or sets of edges, as shown in Fig.

Original: A1.
Revised: A1.

Original: As is typical in DBN we assume that the graph is invariant across different timesteps.
Revised: As is typical in DBN we assume that the graph is invariant across different timesteps.

Original: We assume θ k are constant across the different timesteps.
Revised: We assume θ k are constant across the different timesteps.

Original: We add to the image also the cumulative reward R.
Revised: We add to the image also the cumulative reward R.

Original: In practice we will focus instead on the cumulative future reward R t +1 =
Revised: In practice we will focus instead on the cumulative future reward R t +1 =

Original: (cid:80) T τ =
Revised: (cid:80) T τ =

Original: t +1 γ τ − t − 1 r τ , which only considers the contributions of the r τ in the future with respect to the current timestep t .
Revised: t +1 γ τ − t − 1 r τ , which only considers the contributions of the r τ in the future with respect to the current timestep t .

Original: In order to prove Proposition 1, we ﬁrst need to prove that the compact shared representations s mint and compact shared representations θ mink are all the state and change factors dimensions, respectively, that are conditionally independent of a t given the future cumulative reward R t +1 , even given all other variables:
Revised: In order to prove Proposition 1, we ﬁrst need to prove that the compact shared representations s mint and compact shared representations θ mink are all the state and change factors dimensions, respectively, that are conditionally independent of a t given the future cumulative reward R t +1 , even given all other variables:

Original: A the assumption the
Revised: A the assumption the

Original: G
Revised: G

Original: Markov data a s
Revised: Markov data a s

Original: ∈ s t is s mint
Revised: ∈ s t is s mint

Original: [Equation]
Revised: [Equation]

Original: Similarly, a change factor dimension θ i,k ∈ θ k is part of θ mink iff:
Revised: Similarly, a change factor dimension θ i,k ∈ θ k is part of θ mink iff:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We split the proof in two parts, the "only if" and the "if" part: “If conditionally dependent on a t given R t +1 then in compact representation ":
Revised: We split the proof in two parts, the "only if" and the "if" part: “If conditionally dependent on a t given R t +1 then in compact representation ":

Original: We ﬁrst show that if s i,t satisﬁes the conditional dependence s i,t (cid:54)⊥⊥ a t | R t +1 , ˜ s t , ∀ ˜ s t ⊆ { s t \ s i,t } , then it is part of s mint , i.e. s i,t either has an edge to the reward in the next time-step r t +1 , or, recursively, it has an edge to another state component in the next time-step s j,t +1 , such that the same component at time step t , s j,t +1 ∈ s mint .
Revised: We ﬁrst show that if s i,t satisﬁes the conditional dependence s i,t (cid:54)⊥⊥ a t | R t +1 , ˜ s t , ∀ ˜ s t ⊆ { s t \ s i,t } , then it is part of s mint , i.e. s i,t either has an edge to the reward in the next time-step r t +1 , or, recursively, it has an edge to another state component in the next time-step s j,t +1 , such that the same component at time step t , s j,t +1 ∈ s mint .

Original: Note that this recursive deﬁnition collects all of the states s i,t that have an effect on future reward r t
Revised: Note that this recursive deﬁnition collects all of the states s i,t that have an effect on future reward r t

Original: + τ , τ = { 1 , . .
Revised: + τ , τ = { 1 , . .

Original: , T − t } , either directly, or through the inﬂuence on other state components.
Revised: , T − t } , either directly, or through the inﬂuence on other state components.

Original: Since all of these r t + τ are inﬂuencing the cumulative future reward R t +1 , all of these components have an edge to R t +1 as well.
Revised: Since all of these r t + τ are inﬂuencing the cumulative future reward R t +1 , all of these components have an edge to R t +1 as well.

Original: We prove it by contradiction.
Revised: We prove it by contradiction.

Original: Suppose that s i,t (cid:54)⊥⊥ a t | R t +1 does not have a direct or indirect path to r t
Revised: Suppose that s i,t (cid:54)⊥⊥ a t | R t +1 does not have a direct or indirect path to r t

Original: + τ , i.e. s i,t ∈ s min .
Revised: + τ , i.e. s i,t ∈ s min .

Original: By assumption a t only affects future state s t +1 , so s i,t ⊥⊥ a t .
Revised: By assumption a t only affects future state s t +1 , so s i,t ⊥⊥ a t .

Original: Then, according to the Markov and faithfulness conditions, s i,t is independent of a t conditioning on R t +1 , since there is no path that connects s i,t → · · · → r t
Revised: Then, according to the Markov and faithfulness conditions, s i,t is independent of a t conditioning on R t +1 , since there is no path that connects s i,t → · · · → r t

Original: + τ → R t +1 ← r t +1 ← a t on which R t +1 is a collider (i.e. a variable with two incoming edges), which is the only path which could introduce a conditional dependence.
Revised: + τ → R t +1 ← r t +1 ← a t on which R t +1 is a collider (i.e. a variable with two incoming edges), which is the only path which could introduce a conditional dependence.

Original: This contradicts the assumption.
Revised: This contradicts the assumption.

Original: Similarly we show that if ∀ ˜ l t ⊆ { s t , { θ k \ θ i,k }} the change factor dimension θ i,k (cid:54)⊥⊥ a t | R t +1 , ˜ l t , then θ i,k ∈ θ mink , which similarly to previous case means it has a direct or indirect effect on r t
Revised: Similarly we show that if ∀ ˜ l t ⊆ { s t , { θ k \ θ i,k }} the change factor dimension θ i,k (cid:54)⊥⊥ a t | R t +1 , ˜ l t , then θ i,k ∈ θ mink , which similarly to previous case means it has a direct or indirect effect on r t

Original: + τ and therefore R t +1 .
Revised: + τ and therefore R t +1 .

Original: By contradiction suppose that θ i,k (cid:54)⊥⊥ a t | R t +1 , ˜ l t for all previously deﬁned ˜ l t , but it is not a change parameter for the reward function θ i,k (cid:54)∈ θ rk with c θ k (cid:41) r = 1 , nor it is a change parameter for the state dynamics θ i,k ∈ θ
Revised: By contradiction suppose that θ i,k (cid:54)⊥⊥ a t | R t +1 , ˜ l t for all previously deﬁned ˜ l t , but it is not a change parameter for the reward function θ i,k (cid:54)∈ θ rk with c θ k (cid:41) r = 1 , nor it is a change parameter for the state dynamics θ i,k ∈ θ

Original: s k with a direct or indirect path to r t + τ for τ = 1 , . .
Revised: s k with a direct or indirect path to r t + τ for τ = 1 , . .

Original: , T − t .
Revised: , T − t .

Original: By assumption of our model, θ i,k is never connected to a t directly, nor they might have a common cause, so θ
Revised: By assumption of our model, θ i,k is never connected to a t directly, nor they might have a common cause, so θ

Original: i,k ⊥⊥ a t .
Revised: i,k ⊥⊥ a t .

Original: Then, according to the Markov condition, θ i,k is independent of a t conditioning on R t +1 , which contradicts the assumption, since:
Revised: Then, according to the Markov condition, θ i,k is independent of a t conditioning on R t +1 , which contradicts the assumption, since:

Original: • if θ i,k (cid:54)∈ θ rk
Revised: • if θ i,k (cid:54)∈ θ rk

Original: or c θ k (cid:41) r = 0 , then it means there is no path θ i,k
Revised: or c θ k (cid:41) r = 0 , then it means there is no path θ i,k

Original: → s i,t + τ → r t + τ → R t +1 ←
Revised: → s i,t + τ → r t + τ → R t +1 ←

Original: • if θ i,k ∈ θ sk
Revised: • if θ i,k ∈ θ sk

Original: but there is no directed path to r t
Revised: but there is no directed path to r t

Original: + τ for any τ ≥ 1 , i.e. then there is also no directed path θ i,k → · · · → r t
Revised: + τ for any τ ≥ 1 , i.e. then there is also no directed path θ i,k → · · · → r t

Original: + τ → R t +1 ← r t +1 ← a t that would be open when we condition on R t
Revised: + τ → R t +1 ← r t +1 ← a t that would be open when we condition on R t

Original: +1 .
Revised: +1 .

Original: “If in compact representation then conditionally dependent on a t given R t +1 ":
Revised: “If in compact representation then conditionally dependent on a t given R t +1 ":

Original: We next show that if s i,t ∈ s mint , which mean s i,t has a direct or indirect edge to r t + τ , τ = { 1 , . . .
Revised: We next show that if s i,t ∈ s mint , which mean s i,t has a direct or indirect edge to r t + τ , τ = { 1 , . . .

Original: , T − t } , then s i,t satisﬁes the conditional dependence s i,t (cid:54)⊥⊥ a t | R t +1 ,
Revised: , T − t } , then s i,t satisﬁes the conditional dependence s i,t (cid:54)⊥⊥ a t | R t +1 ,

Original: ˜ s t , ∀ ˜ s t ⊆ { s t \ s i,t } .
Revised: ˜ s t , ∀ ˜ s t ⊆ { s t \ s i,t } .

Original: We prove it by contradiction.
Revised: We prove it by contradiction.

Original: Suppose s i,t has a directed path to r t
Revised: Suppose s i,t has a directed path to r t

Original: + τ and s i,t is independent on a t given R t +1 and a subset of other variables ˜ s t ⊆
Revised: + τ and s i,t is independent on a t given R t +1 and a subset of other variables ˜ s t ⊆

Original: s t \ s i,t .
Revised: s t \ s i,t .

Original: Since we assume that there are no instantaneous causal relations across the state dimensions, if s i,t (cid:54)⊥⊥ a t | R t +1 there can never be an s j,t such that s i,t ⊥⊥ a t | R t +1 , s j,t .
Revised: Since we assume that there are no instantaneous causal relations across the state dimensions, if s i,t (cid:54)⊥⊥ a t | R t +1 there can never be an s j,t such that s i,t ⊥⊥ a t | R t +1 , s j,t .

Original: In this case, this means that s i,t ⊥⊥ a t | R t +1 has to hold.
Revised: In this case, this means that s i,t ⊥⊥ a t | R t +1 has to hold.

Original: Then according to the Markov and faithfulness assumptions, s i,t cannot have any directed path to any r t + τ ∀ τ ≥ 1 , because that any such path create a v-structure in the collider R t +1 , which would be open if we condition on R t +1 , contradicting the assumption.
Revised: Then according to the Markov and faithfulness assumptions, s i,t cannot have any directed path to any r t + τ ∀ τ ≥ 1 , because that any such path create a v-structure in the collider R t +1 , which would be open if we condition on R t +1 , contradicting the assumption.

Original: Similarly, suppose θ i,k is a dimension in θ k that has a directed path to r t
Revised: Similarly, suppose θ i,k is a dimension in θ k that has a directed path to r t

Original: + τ .
Revised: + τ .

Original: We distinguish two cases, and show in neither can θ i,k be independent of a t given R t +1 and a subset of the other variables:
Revised: We distinguish two cases, and show in neither can θ i,k be independent of a t given R t +1 and a subset of the other variables:

Original: • if θ i,k ∈ θ rk , then it cannot be independent of a t when we condition on R t +1 , which is a descendant of r t +1 and therefore opens the collider path θ rk ← r t +1 ← a t ; • if θ i,k ∈ θ sk , then at timestep t it is always only connected to the corresponding s i,t .
Revised: • if θ i,k ∈ θ rk , then it cannot be independent of a t when we condition on R t +1 , which is a descendant of r t +1 and therefore opens the collider path θ rk ← r t +1 ← a t ; • if θ i,k ∈ θ sk , then at timestep t it is always only connected to the corresponding s i,t .

Original: So if there is a directed path π to r t + τ , it has to go through s i,t .
Revised: So if there is a directed path π to r t + τ , it has to go through s i,t .

Original: While π cannot be blocked by any subset of { θ k \ θ i,k } , it can be blocked by conditioning on s i,t , there are inﬁnite future paths with the same structure, e.g. through s i,t +1 that will not be blocked by conditioning only on variables at timestep t .
Revised: While π cannot be blocked by any subset of { θ k \ θ i,k } , it can be blocked by conditioning on s i,t , there are inﬁnite future paths with the same structure, e.g. through s i,t +1 that will not be blocked by conditioning only on variables at timestep t .

Original: Under the faithfulness and Markov assumption this means that θ i,k cannot be independent from a t by conditioning on any subset of state variables at timestep t or any other change parameters, which is a contradiction.
Revised: Under the faithfulness and Markov assumption this means that θ i,k cannot be independent from a t by conditioning on any subset of state variables at timestep t or any other change parameters, which is a contradiction.

Original: We can now prove our main proposition:
Revised: We can now prove our main proposition:

Original: Proposition 1.
Revised: Proposition 1.

Original: Under the assumption that the graph G is Markov and faithful to the measured data, the union of compact domain-speciﬁc θ mink and compact shared representations s mint are the minimal and sufﬁcient dimensions for policy learning across domains.
Revised: Under the assumption that the graph G is Markov and faithful to the measured data, the union of compact domain-speciﬁc θ mink and compact shared representations s mint are the minimal and sufﬁcient dimensions for policy learning across domains.

Original: Proof.
Revised: Proof.

Original: As shown in the previous lemma, in compact domain-generalizable representations s mint every dimension is dependent on a t given R t +1 and any other variables, and every other dimension is independent of a t given R t
Revised: As shown in the previous lemma, in compact domain-generalizable representations s mint every dimension is dependent on a t given R t +1 and any other variables, and every other dimension is independent of a t given R t

Original: +1 and some other variables.
Revised: +1 and some other variables.

Original: Furthermore, because every dimension that is dependent on a t is necessary for the policy learning and every dimension that is (conditionally) independent of a t for at least a subset of other variables is not necessary for the policy learning, compact domain-speciﬁc θ mink and compact shared representations s mint contain minimal and sufﬁcient dimensions for policy learning across domains.
Revised: Furthermore, because every dimension that is dependent on a t is necessary for the policy learning and every dimension that is (conditionally) independent of a t for at least a subset of other variables is not necessary for the policy learning, compact domain-speciﬁc θ mink and compact shared representations s mint contain minimal and sufﬁcient dimensions for policy learning across domains.

Original: Note that the agents determine the action under the condition of maximizing cumulative reward, which policy learning aims to achieve, so we always consider the situation when the discounted cumulative future reward R t +1 is given.
Revised: Note that the agents determine the action under the condition of maximizing cumulative reward, which policy learning aims to achieve, so we always consider the situation when the discounted cumulative future reward R t +1 is given.

Original: A P ROOF OF T HEOREM
Revised: A P ROOF OF T HEOREM

Original: Theorem 1 (Structural Identiﬁability) .
Revised: Theorem 1 (Structural Identiﬁability) .

Original: Suppose the underlying states s t are observed, i.e., Eq.
Revised: Suppose the underlying states s t are observed, i.e., Eq.

Original: (1) is an MDP.
Revised: (1) is an MDP.

Original: Then under the Markov condition and faithfulness assumption, the structural matrices
Revised: Then under the Markov condition and faithfulness assumption, the structural matrices

Original: C s (cid:41) s , c a (cid:41) s , c s (cid:41) r , c a (cid:41) r , C θ k (cid:41) s , and c θ k (cid:41) r are identiﬁable .
Revised: C s (cid:41) s , c a (cid:41) s , c s (cid:41) r , c a (cid:41) r , C θ k (cid:41) s , and c θ k (cid:41) r are identiﬁable .

Original: Proof.
Revised: Proof.

Original: We concatenate data from different domains and denote by k be the variable that takes the domain index 1 , · · · , n .
Revised: We concatenate data from different domains and denote by k be the variable that takes the domain index 1 , · · · , n .

Original: Since the data distribution changes across domains and the change is due to the unobserved change factors θ k that inﬂuence the observed variables, we can represent the change factors as a function of k .
Revised: Since the data distribution changes across domains and the change is due to the unobserved change factors θ k that inﬂuence the observed variables, we can represent the change factors as a function of k .

Original: In other words, we use the domain index k as a surrogate variable to characterize the unobserved change factors.
Revised: In other words, we use the domain index k as a surrogate variable to characterize the unobserved change factors.

Original: [Figure]
Revised: [Figure]

Original: We denote the variable set in the system by V , with V =
Revised: We denote the variable set in the system by V , with V =

Original: { s 1 ,t − 1 , . . .
Revised: { s 1 ,t − 1 , . . .

Original: , s d,t − 1 , s 1 ,t , . . .
Revised: , s d,t − 1 , s 1 ,t , . . .

Original: , s d,t , a t − 1 , r t } , and the variables form a dynamic Bayesian network
Revised: , s d,t , a t − 1 , r t } , and the variables form a dynamic Bayesian network

Original: G .
Revised: G .

Original: Note that in our particular problem, according to the generative environment model in Eq.
Revised: Note that in our particular problem, according to the generative environment model in Eq.

Original: (1), the possible edges in G are only those from s i,t − 1 ∈ s t − 1 to s j,t ∈ s t , from a t − 1 to s j,t ∈ s t , from s i,t − 1 ∈ s t − 1 to r t , and from a t − 1 to r t .
Revised: (1), the possible edges in G are only those from s i,t − 1 ∈ s t − 1 to s j,t ∈ s t , from a t − 1 to s j,t ∈ s t , from s i,t − 1 ∈ s t − 1 to r t , and from a t − 1 to r t .

Original: We further include the domain index k into the system to characterize the unobserved change factors.
Revised: We further include the domain index k into the system to characterize the unobserved change factors.

Original: It has been shown that under the Markov condition and faithfulness assumption, for every V i , V j ∈ V ,
Revised: It has been shown that under the Markov condition and faithfulness assumption, for every V i , V j ∈ V ,

Original: V i and V j are not adjacent in G if
Revised: V i and V j are not adjacent in G if

Original: and only if they are independent conditional on some subset of { V l | l (cid:54)
Revised: and only if they are independent conditional on some subset of { V l | l (cid:54)

Original: =
Revised: =

Original: i, l (cid:54) =
Revised: i, l (cid:54) =

Original: j } ∪ k (Huang et al., 2020).
Revised: j } ∪ k (Huang et al., 2020).

Original: Thus, we can asymptotically identity the correct graph skeleton over V .
Revised: Thus, we can asymptotically identity the correct graph skeleton over V .

Original: Moreover, since we assume a dynamic Bayesian network, there the direction of an edge between a variable at time t to one at time t + 1 is ﬁxed.
Revised: Moreover, since we assume a dynamic Bayesian network, there the direction of an edge between a variable at time t to one at time t + 1 is ﬁxed.

Original: Therefore, the structural matrices C s (cid:41) s , c a (cid:41) s , c s (cid:41) r , and c a (cid:41) r , which are parts of the graph G over V , are identiﬁable.
Revised: Therefore, the structural matrices C s (cid:41) s , c a (cid:41) s , c s (cid:41) r , and c a (cid:41) r , which are parts of the graph G over V , are identiﬁable.

Original: Furthermore, we want to show the identiﬁability of C θ k (cid:41) s , and c θ k (cid:41) r ; that is, to identify which distribution modules have changes.
Revised: Furthermore, we want to show the identiﬁability of C θ k (cid:41) s , and c θ k (cid:41) r ; that is, to identify which distribution modules have changes.

Original: Whether a variable V i has a changing module is decided by whether V i and k are independent conditional on some subset of other variables.
Revised: Whether a variable V i has a changing module is decided by whether V i and k are independent conditional on some subset of other variables.

Original: The justiﬁcation for one side of this decision is trivial.
Revised: The justiﬁcation for one side of this decision is trivial.

Original: If V i ’s module does not change, that means P ( V i | PA i ) remains the same for every value of k , and so V i ⊥⊥ k | PA i .
Revised: If V i ’s module does not change, that means P ( V i | PA i ) remains the same for every value of k , and so V i ⊥⊥ k | PA i .

Original: Thus, if V i and k are not independent conditional on any subset of other variables, V i ’s module changes with k , which is represented by an edge between
Revised: Thus, if V i and k are not independent conditional on any subset of other variables, V i ’s module changes with k , which is represented by an edge between

Original: V i and k .
Revised: V i and k .

Original: Conversely, we assume that if V i ’s module changes, which entails that V i and k are not independent given PA i , then V i and k are not independent given any other subset of V \{ V i } .
Revised: Conversely, we assume that if V i ’s module changes, which entails that V i and k are not independent given PA i , then V i and k are not independent given any other subset of V \{ V i } .

Original: If this assumption does not hold, then we only claim to detect some (but not necessarily all) variables with changing modules.
Revised: If this assumption does not hold, then we only claim to detect some (but not necessarily all) variables with changing modules.

Original: A P ROOF OF T HEOREM
Revised: A P ROOF OF T HEOREM

Original: In this section, we derive the generalization bound under the PAC-Bayes framework (McAllester, 1999; Shalev-Shwartz and Ben-David, 2014), and our formulation follows (Pentina and Lampert, 2014) and (Amit and Meir, 2018).
Revised: In this section, we derive the generalization bound under the PAC-Bayes framework (McAllester, 1999; Shalev-Shwartz and Ben-David, 2014), and our formulation follows (Pentina and Lampert, 2014) and (Amit and Meir, 2018).

Original: We assume that all domains share the sample space Z , hypothesis space H , and loss function (cid:96) : H×Z → [0 , 1] .
Revised: We assume that all domains share the sample space Z , hypothesis space H , and loss function (cid:96) : H×Z → [0 , 1] .

Original: All domains differ in the unknown sample distribution
Revised: All domains differ in the unknown sample distribution

Original: E k parameterized by θ min k associated with each domain k .
Revised: E k parameterized by θ min k associated with each domain k .

Original: We observe the training sets S 1 , . . .
Revised: We observe the training sets S 1 , . . .

Original: , S n corresponding to n different domains.
Revised: , S n corresponding to n different domains.

Original: The number of samples in domain k is denoted by m k .
Revised: The number of samples in domain k is denoted by m k .

Original: Each dataset S k is assumed to be generated from an unknown sample distribution
Revised: Each dataset S k is assumed to be generated from an unknown sample distribution

Original: E m k k .
Revised: E m k k .

Original: We also assume that the sample distribution E k are generated i.i.d.
Revised: We also assume that the sample distribution E k are generated i.i.d.

Original: from an unknown domain distribution τ .
Revised: from an unknown domain distribution τ .

Original: More speciﬁcally, we have S k
Revised: More speciﬁcally, we have S k

Original: = ( z 1 ,k , . . .
Revised: = ( z 1 ,k , . . .

Original: , z i,k , . . . ,
Revised: , z i,k , . . . ,

Original: z m k ,k ) , where z i,k = ( s i,k , v ∗ ( s i,k )) .
Revised: z m k ,k ) , where z i,k = ( s i,k , v ∗ ( s i,k )) .

Original: Note that, s i,k is the i -th state sampled from k -th domain and v ∗ ( s i,k ) is its corresponding optimal state-value.
Revised: Note that, s i,k is the i -th state sampled from k -th domain and v ∗ ( s i,k ) is its corresponding optimal state-value.

Original: For any value function h θ min k ( · ) parameterized by θ min k , we deﬁne the loss function (cid:96) ( h θ min k , z
Revised: For any value function h θ min k ( · ) parameterized by θ min k , we deﬁne the loss function (cid:96) ( h θ min k , z

Original: i,k ) =
Revised: i,k ) =

Original: D dist ( h θ min k
Revised: D dist ( h θ min k

Original: ( s i,k ) , v ∗ ( s i,k )) , where D dist is a distance function that measures the discrepancy between the learned value and the optimal state-value.
Revised: ( s i,k ) , v ∗ ( s i,k )) , where D dist is a distance function that measures the discrepancy between the learned value and the optimal state-value.

Original: We also let P be the prior distribution over H and Q the posterior over H .
Revised: We also let P be the prior distribution over H and Q the posterior over H .

Original: Theorem 1.
Revised: Theorem 1.

Original: Let Q be an arbitrary distribution over θ mink and P the prior distribution over θ mink .
Revised: Let Q be an arbitrary distribution over θ mink and P the prior distribution over θ mink .

Original: Then for any δ ∈ (0 , 1] , with probability at least 1 − δ , the following inequality holds uniformly for all Q ,
Revised: Then for any δ ∈ (0 , 1] , with probability at least 1 − δ , the following inequality holds uniformly for all Q ,

Original: [Equation]
Revised: [Equation]

Original: where er ( Q ) and ˆ er ( Q , S k ) are the generalization error and the training error between the estimated value and the optimal value, respectively.
Revised: where er ( Q ) and ˆ er ( Q , S k ) are the generalization error and the training error between the estimated value and the optimal value, respectively.

Original: Proof.
Revised: Proof.

Original: This proof consists of two steps, both using the classical PAC-Bayes bound (McAllester, 1999;
Revised: This proof consists of two steps, both using the classical PAC-Bayes bound (McAllester, 1999;

Original: Shalev-Shwartz and Ben-David, 2014).
Revised: Shalev-Shwartz and Ben-David, 2014).

Original: Therefore, we start by restating the classical PCA-Bayes bound.
Revised: Therefore, we start by restating the classical PCA-Bayes bound.

Original: Theorem A1 (Classical PAC-Bayes Bound, General Notations) .
Revised: Theorem A1 (Classical PAC-Bayes Bound, General Notations) .

Original: Let X be a sample space, P ( X ) a distribution over X , Θ a hypothesis space.
Revised: Let X be a sample space, P ( X ) a distribution over X , Θ a hypothesis space.

Original: Given a loss function (cid:96) ( θ, X ) :
Revised: Given a loss function (cid:96) ( θ, X ) :

Original: Θ × X → [0 , 1] and a collection of M i.i.d random variables ( X 1 , . . .
Revised: Θ × X → [0 , 1] and a collection of M i.i.d random variables ( X 1 , . . .

Original: , X M ) sampled from P ( X ) , let π be a prior distribution over hypothesis in Θ .
Revised: , X M ) sampled from P ( X ) , let π be a prior distribution over hypothesis in Θ .

Original: Then, for any δ ∈ (0 , 1] , the following bound holds uniformly for all posterior distributions ρ over Θ ,
Revised: Then, for any δ ∈ (0 , 1] , the following bound holds uniformly for all posterior distributions ρ over Θ ,

Original: [Equation]
Revised: [Equation]

Original: Between-domain Generalization Bound First, we bound the between-domain generalization, i.e., relating er ( Q ) to er ( Q , E k ) .
Revised: Between-domain Generalization Bound First, we bound the between-domain generalization, i.e., relating er ( Q ) to er ( Q , E k ) .

Original: We ﬁrst expand the generalization error as below,
Revised: We ﬁrst expand the generalization error as below,

Original: [Equation]
Revised: [Equation]

Original: Then we compute the error across the training domains,
Revised: Then we compute the error across the training domains,

Original: [Equation]
Revised: [Equation]

Original: Then Theorem A1 says that for any δ 0 ∼ (0 , 1] , we have
Revised: Then Theorem A1 says that for any δ 0 ∼ (0 , 1] , we have

Original: [Equation]
Revised: [Equation]

Original: where P is a prior distribution over θ .
Revised: where P is a prior distribution over θ .

Original: Within-domain Generalization Bound
Revised: Within-domain Generalization Bound

Original: Then, we bound the the within-domain generalization,
Revised: Then, we bound the the within-domain generalization,

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e., relating er ( Q , E k ) to ˆ er ( Q , S k ) .
Revised: e., relating er ( Q , E k ) to ˆ er ( Q , S k ) .

Original: We ﬁrst have
Revised: We ﬁrst have

Original: [Equation]
Revised: [Equation]

Original: Then we compute the empirical error across the training domains,
Revised: Then we compute the empirical error across the training domains,

Original: [Equation]
Revised: [Equation]

Original: According to Theorem A1, for any δ D ∼ (0 , 1] , we have
Revised: According to Theorem A1, for any δ D ∼ (0 , 1] , we have

Original: [Equation]
Revised: [Equation]

Original: With the choice of π = (cid:82) P ( θ ) Q ( S D , θ ) d θ and ρ = (cid:82) Q ( θ ) Q ( S D , θ ) d θ , we have that
Revised: With the choice of π = (cid:82) P ( θ ) Q ( S D , θ ) d θ and ρ = (cid:82) Q ( θ ) Q ( S D , θ ) d θ , we have that

Original: D KL ( ρ || π ) ≤ D KL ( Q||P )
Revised: D KL ( ρ || π ) ≤ D KL ( Q||P )

Original: (Yin et al., 2019).
Revised: (Yin et al., 2019).

Original: Thus, the above inequality can be further written as,
Revised: Thus, the above inequality can be further written as,

Original: [Equation]
Revised: [Equation]

Original: Overall Generalization Bound Combining Eq.
Revised: Overall Generalization Bound Combining Eq.

Original: (A3) and (A7) using the union bound and choosing that for any δ > 0 , set δ 0 .
Revised: (A3) and (A7) using the union bound and choosing that for any δ > 0 , set δ 0 .

Original: =
Revised: =

Original: δ 2 and δ k .
Revised: δ 2 and δ k .

Original: =
Revised: =

Original: δ 2 n for k = 1 , . . . , n , then we ﬁnally obtain,
Revised: δ 2 n for k = 1 , . . . , n , then we ﬁnally obtain,

Original: [Equation]
Revised: [Equation]

Original: A M ORE DETAILS FOR MODEL ESTIMATION A4. L OCATING MODEL CHANGES
Revised: A M ORE DETAILS FOR MODEL ESTIMATION A4. L OCATING MODEL CHANGES

Original: In real-world scenarios, it is often the case that changes to the environment are sparse and localized.
Revised: In real-world scenarios, it is often the case that changes to the environment are sparse and localized.

Original: Instead of assuming every function to change arbitrarily, which is inefﬁcient and unnecessarily complex, we ﬁrst identify possible locations of the changes.
Revised: Instead of assuming every function to change arbitrarily, which is inefﬁcient and unnecessarily complex, we ﬁrst identify possible locations of the changes.

Original: To this end, we concatenate data from different domains and denote by k the variable that takes distinct values 1 , · · · , n to represent the domain index.
Revised: To this end, we concatenate data from different domains and denote by k the variable that takes distinct values 1 , · · · , n to represent the domain index.

Original: Then, we exploit (conditional) independencies/dependencies to locate model changes.
Revised: Then, we exploit (conditional) independencies/dependencies to locate model changes.

Original: These (conditional) independencies/dependencies can be tested by kernel-based conditional independence tests (Zhang et al., 2011), which allows for both linear or nonlinear relationships between variables.
Revised: These (conditional) independencies/dependencies can be tested by kernel-based conditional independence tests (Zhang et al., 2011), which allows for both linear or nonlinear relationships between variables.

Original: Below we show that in some cases, we can identify the location of θ k , by using the conditional independence relationships from concatenated data.
Revised: Below we show that in some cases, we can identify the location of θ k , by using the conditional independence relationships from concatenated data.

Original: Proposition A1.
Revised: Proposition A1.

Original: In POMDP, where the underlying states are latent, we can localize the changes by conditional independence relationships from concatenated observed data D in the following cases:
Revised: In POMDP, where the underlying states are latent, we can localize the changes by conditional independence relationships from concatenated observed data D in the following cases:

Original: C 1 : if o t ⊥⊥ k , then there is neither a change in the observation function nor in the state dynamics for any state that is an input to the observation function; C 2 : if o t ⊥⊥ k and a t (cid:54)⊥⊥ k | r t +1 , then there is only a change in the reward function; C 3 : if a t ⊥⊥ k | r t +1 , then there is neither a change in the reward function nor in the state dynamics for any state in s min ; C 4 : if a t ⊥⊥ k | r t +1 and o t (cid:54)⊥⊥ k , then there is a change in the observation function, or there exists a state that is not in s min but is an input to the observation function, whose dynamics has a change.
Revised: C 1 : if o t ⊥⊥ k , then there is neither a change in the observation function nor in the state dynamics for any state that is an input to the observation function; C 2 : if o t ⊥⊥ k and a t (cid:54)⊥⊥ k | r t +1 , then there is only a change in the reward function; C 3 : if a t ⊥⊥ k | r t +1 , then there is neither a change in the reward function nor in the state dynamics for any state in s min ; C 4 : if a t ⊥⊥ k | r t +1 and o t (cid:54)⊥⊥ k , then there is a change in the observation function, or there exists a state that is not in s min but is an input to the observation function, whose dynamics has a change.

Original: Proof.
Revised: Proof.

Original: We ﬁrst formulate the problem as follows.
Revised: We ﬁrst formulate the problem as follows.

Original: We concatenate data from different domains and use domain-index variable k to indicate whether the corresponding distribution module has changes across domains.
Revised: We concatenate data from different domains and use domain-index variable k to indicate whether the corresponding distribution module has changes across domains.

Original: Speciﬁcally, by assuming the Markov condition and faithfulness assumption, s i,t has an edge with k if and only if p ( s i,t | PA ( s i,t )) changes across domains, where PA ( · ) denotes its parents.
Revised: Speciﬁcally, by assuming the Markov condition and faithfulness assumption, s i,t has an edge with k if and only if p ( s i,t | PA ( s i,t )) changes across domains, where PA ( · ) denotes its parents.

Original: Similarly, r t has an edge with k if and only
Revised: Similarly, r t has an edge with k if and only

Original: if p ( r t | PA ( r t )) changes across domains, and o t has an edge with k
Revised: if p ( r t | PA ( r t )) changes across domains, and o t has an edge with k

Original: if and only if p ( o t | PA ( o t )) changes across domains.
Revised: if and only if p ( o t | PA ( o t )) changes across domains.

Original: Under this setting, locating changes is equivalent to identify which variables have an edge with k from the data.
Revised: Under this setting, locating changes is equivalent to identify which variables have an edge with k from the data.

Original: We consider the scenario of POMDP, where we only observe { o t , r t , a t } and the underlying states s t are not observed.
Revised: We consider the scenario of POMDP, where we only observe { o t , r t , a t } and the underlying states s t are not observed.

Original: Below, we consider each case separately.
Revised: Below, we consider each case separately.

Original: Case 1: Show that if o t ⊥⊥ k , then there is neither a change in the observation function nor a change in the state dynamics for any state that is an input to the observation function.
Revised: Case 1: Show that if o t ⊥⊥ k , then there is neither a change in the observation function nor a change in the state dynamics for any state that is an input to the observation function.

Original: We prove it by contradiction.
Revised: We prove it by contradiction.

Original: Suppose that there is a change in the observation function and a change in the state dynamics for any state that is an input to the observation function.
Revised: Suppose that there is a change in the observation function and a change in the state dynamics for any state that is an input to the observation function.

Original: That is, o t has an edge with k , and s i,t that has a direct edge to o t also connects with k .
Revised: That is, o t has an edge with k , and s i,t that has a direct edge to o t also connects with k .

Original: Based on faithfulness assumption, o t (cid:54)⊥⊥ k , which contradicts to the assumption.
Revised: Based on faithfulness assumption, o t (cid:54)⊥⊥ k , which contradicts to the assumption.

Original: Since we have a contradiction, it must be that there is neither a change in the observation function nor a change in the state dynamics for any state that is an input to the observation function.
Revised: Since we have a contradiction, it must be that there is neither a change in the observation function nor a change in the state dynamics for any state that is an input to the observation function.

Original: Case 2: Show that if o t ⊥⊥ k and a t (cid:54)⊥⊥ k | r t +1 , then there is only a change in the reward function.
Revised: Case 2: Show that if o t ⊥⊥ k and a t (cid:54)⊥⊥ k | r t +1 , then there is only a change in the reward function.

Original: If o t ⊥⊥ k and a t − 1 (cid:54)⊥⊥ k | r t , based on the Markov condition and faithfulness assumption, r t has an edge with k , and s i,t and o t do not have edges with k ; that is, there are only changes in the reward function.
Revised: If o t ⊥⊥ k and a t − 1 (cid:54)⊥⊥ k | r t , based on the Markov condition and faithfulness assumption, r t has an edge with k , and s i,t and o t do not have edges with k ; that is, there are only changes in the reward function.

Original: Case 3: Show that if a t ⊥⊥ k | r t +1 , then there is neither a change in the reward function nor a change in the state dynamics for any state in s min .
Revised: Case 3: Show that if a t ⊥⊥ k | r t +1 , then there is neither a change in the reward function nor a change in the state dynamics for any state in s min .

Original: By contradiction, suppose that there is a change in the reward function or there exists a state s j,t ∈ s minj that has a change in its dynamics.
Revised: By contradiction, suppose that there is a change in the reward function or there exists a state s j,t ∈ s minj that has a change in its dynamics.

Original: That is, r t has an edge with k or corresponding s j,t has an edge with k .
Revised: That is, r t has an edge with k or corresponding s j,t has an edge with k .

Original: Based on faithfulness assumption, a t (cid:54)⊥⊥ k | r t +1 , which contradicts to the assumption.
Revised: Based on faithfulness assumption, a t (cid:54)⊥⊥ k | r t +1 , which contradicts to the assumption.

Original: Case 4: Show that if a t ⊥⊥ k | r t +1 and o t (cid:54)⊥⊥ k , then there is a change in the observation function, or there exists a state that is not in s min but is an input to the observation function, whose dynamics has a change.
Revised: Case 4: Show that if a t ⊥⊥ k | r t +1 and o t (cid:54)⊥⊥ k , then there is a change in the observation function, or there exists a state that is not in s min but is an input to the observation function, whose dynamics has a change.

Original: According to Case 3, if a t ⊥⊥ k | r t +1 , then there is neither a change in the reward function nor a change in the state dynamics for any state in s min .
Revised: According to Case 3, if a t ⊥⊥ k | r t +1 , then there is neither a change in the reward function nor a change in the state dynamics for any state in s min .

Original: Furthermore, since o t (cid:54)⊥⊥ k , then based on the
Revised: Furthermore, since o t (cid:54)⊥⊥ k , then based on the

Original: Markov condition, either o t has an edge with k , or there exists a state that is not in s min but is an input to the observation function, whose dynamics has an edge with k .
Revised: Markov condition, either o t has an edge with k , or there exists a state that is not in s min but is an input to the observation function, whose dynamics has an edge with k .

Original: That is, there is a change in the observation function, or there exists a state that is not in s min but is an input to the observation function, whose dynamics has a change.
Revised: That is, there is a change in the observation function, or there exists a state that is not in s min but is an input to the observation function, whose dynamics has a change.

Original: Based on Theorem 1 and Proposition A1, in MDP, we can fully determine where the changes are, so we only need to consider the corresponding θ ( · ) k to capture the changes.
Revised: Based on Theorem 1 and Proposition A1, in MDP, we can fully determine where the changes are, so we only need to consider the corresponding θ ( · ) k to capture the changes.

Original: In POMDP, in Case 1, we only need to involve θ sk and θ rk in model estimation, that is, θ k
Revised: In POMDP, in Case 1, we only need to involve θ sk and θ rk in model estimation, that is, θ k

Original: = { θ sk , θ rk } ; in Case 2, θ k
Revised: = { θ sk , θ rk } ; in Case 2, θ k

Original: = { θ rk } ; and in Case 3 & 4, θ k
Revised: = { θ rk } ; and in Case 3 & 4, θ k

Original: = { θ ok , θ sk } .
Revised: = { θ ok , θ sk } .

Original: For other cases, we involve θ k
Revised: For other cases, we involve θ k

Original: = { θ ok , θ sk , θ rk } in model estimation.
Revised: = { θ ok , θ sk , θ rk } in model estimation.

Original: A4. M ORE DETAILS FOR ESTIMATION OF DOMAIN - VARYING MODELS IN ONE STEP
Revised: A4. M ORE DETAILS FOR ESTIMATION OF DOMAIN - VARYING MODELS IN ONE STEP

Original: We use MiSS-VAE to learn the environment model, which contains three components: the "sequential
Revised: We use MiSS-VAE to learn the environment model, which contains three components: the "sequential

Original: VAE" component, the "multi-model" component, and the "structure" component.
Revised: VAE" component, the "multi-model" component, and the "structure" component.

Original: Figure A3 gives the diagram of neural network architecture in model training.
Revised: Figure A3 gives the diagram of neural network architecture in model training.

Original: Speciﬁcally, for the "sequential VAE" component, we include a Long Short-Term Memory (LSTM (Hochreiter and Schmidhuber, 1997)) to encode the sequential information with output h t and a
Revised: Speciﬁcally, for the "sequential VAE" component, we include a Long Short-Term Memory (LSTM (Hochreiter and Schmidhuber, 1997)) to encode the sequential information with output h t and a

Original: Mixture Density Network (MDN (Bishop, 1994)) to output the parameters of MoGs, and thus to learn the inference model q φ ( s t,k | s t − 1 ,k , y 1: t,k , a 1: t − 1 ,k ; θ k ) and infer a sample of s t,k from q φ as the output.
Revised: Mixture Density Network (MDN (Bishop, 1994)) to output the parameters of MoGs, and thus to learn the inference model q φ ( s t,k | s t − 1 ,k , y 1: t,k , a 1: t − 1 ,k ; θ k ) and infer a sample of s t,k from q φ as the output.

Original: The generated sample further acts as an input to the decoder, and the decoder outputs ˆ o t
Revised: The generated sample further acts as an input to the decoder, and the decoder outputs ˆ o t

Original: +

Revised: +

Original: and ˆ r t +2 .
Revised: and ˆ r t +2 .

Original: Moreover, the state dynamics which satisﬁes a Markov process is modeled with an
Revised: Moreover, the state dynamics which satisﬁes a Markov process is modeled with an

Original: MLP and MDN.
Revised: MLP and MDN.

Original: For the "multi-model" component, we include the domain index k as an input to LSTM and involve θ k as free parameters in the inference model q φ , by assuming that θ k also characterizes the changes in the inference model.
Revised: For the "multi-model" component, we include the domain index k as an input to LSTM and involve θ k as free parameters in the inference model q φ , by assuming that θ k also characterizes the changes in the inference model.

Original: Moreover, we embed θ sk in state dynamics p γ , θ ok in observation function and θ rk in reward function in the decoder.
Revised: Moreover, we embed θ sk in state dynamics p γ , θ ok in observation function and θ rk in reward function in the decoder.

Original: With such a characterization, except θ k , all other parameters are shared across domains, so that all we need to update in the target domain is the low-dimensional θ k , which greatly improves the sample efﬁciency and the statistical efﬁciency in the target domain–usually few samples are needed.
Revised: With such a characterization, except θ k , all other parameters are shared across domains, so that all we need to update in the target domain is the low-dimensional θ k , which greatly improves the sample efﬁciency and the statistical efﬁciency in the target domain–usually few samples are needed.

Original: For the "structure" component, the latent states are organized with structures, captured by the mask
Revised: For the "structure" component, the latent states are organized with structures, captured by the mask

Original: C s (cid:41) s .
Revised: C s (cid:41) s .

Original: Also, the structural relationships among perceived signals, latent states, the action variable, the reward variable, and the domain-speciﬁc factors are embedded as free parameters (structural vectors and scalars c s (cid:41) s , c a (cid:41) s , c θ k (cid:41) s , c s (cid:41) r , and c a (cid:41) r ) into MiSS-VAE.
Revised: Also, the structural relationships among perceived signals, latent states, the action variable, the reward variable, and the domain-speciﬁc factors are embedded as free parameters (structural vectors and scalars c s (cid:41) s , c a (cid:41) s , c θ k (cid:41) s , c s (cid:41) r , and c a (cid:41) r ) into MiSS-VAE.

Original: A C OMPLETE EXPERIMENTAL RESULTS
Revised: A C OMPLETE EXPERIMENTAL RESULTS

Original: In this section we provide the complete experimental results on both of our settings, modiﬁed Cartpole and modiﬁed Pong in the OpenAI Gym (Brockman et al., 2016).
Revised: In this section we provide the complete experimental results on both of our settings, modiﬁed Cartpole and modiﬁed Pong in the OpenAI Gym (Brockman et al., 2016).

Original: In the POMDP setting, we use images as input, which for Cartpole look like Fig.
Revised: In the POMDP setting, we use images as input, which for Cartpole look like Fig.

Original: A4(a)
Revised: A4(a)

Original: and for Pong look like Fig.
Revised: and for Pong look like Fig.

Original: A8(a).
Revised: A8(a).

Original: For
Revised: For

Original: Cartpole, we also consider the MDP case, where the true states (cart position and velocity, pole angle and angular velocity) are used as the input to the model.
Revised: Cartpole, we also consider the MDP case, where the true states (cart position and velocity, pole angle and angular velocity) are used as the input to the model.

Original: We consider changes in the state dynamics (e.g., the change of gravity or cart mass in Cartpole, or the change of orientation in Atari), changes in perceived signals (e.g., different noise levels on observed images in Cartpole, as shown in Fig.
Revised: We consider changes in the state dynamics (e.g., the change of gravity or cart mass in Cartpole, or the change of orientation in Atari), changes in perceived signals (e.g., different noise levels on observed images in Cartpole, as shown in Fig.

Original: A4 or colors in Pong) and changes in reward functions (e.g., different reward functions in Pong based on the contact point of the ball), as shown in Fig.
Revised: A4 or colors in Pong) and changes in reward functions (e.g., different reward functions in Pong based on the contact point of the ball), as shown in Fig.

Original: A8 for
Revised: A8 for

Original: Pong.
Revised: Pong.

Original: For each of these factors, we take into account both interpolation (where the factor value in the target domain is in the support of that in source domains), and extrapolation (where it is out of the support w.r.t.
Revised: For each of these factors, we take into account both interpolation (where the factor value in the target domain is in the support of that in source domains), and extrapolation (where it is out of the support w.r.t.

Original: the source domains).
Revised: the source domains).

Original: We train on n source domains based on the trajectory data generated by a random policy.
Revised: We train on n source domains based on the trajectory data generated by a random policy.

Original: In Cartpole experiments, for each domain we collect 10000 trials with 40 steps.
Revised: In Cartpole experiments, for each domain we collect 10000 trials with 40 steps.

Original: For Pong experiments, each domain contains 40 episodes data and each of them takes a maximum of 10000 steps.
Revised: For Pong experiments, each domain contains 40 episodes data and each of them takes a maximum of 10000 steps.

Original: A5. C OMPLETE RESULTS OF MODIFIED C ARTPOLE EXPERIMENT
Revised: A5. C OMPLETE RESULTS OF MODIFIED C ARTPOLE EXPERIMENT

Original: The Cartpole problem consists of a cart and a vertical pendulum attached to the cart using a passive pivot joint.
Revised: The Cartpole problem consists of a cart and a vertical pendulum attached to the cart using a passive pivot joint.

Original: The cart can move left or right.
Revised: The cart can move left or right.

Original: The task is to prevent the vertical pendulum from falling by putting a force on the cart to move it left or right.
Revised: The task is to prevent the vertical pendulum from falling by putting a force on the cart to move it left or right.

Original: The action space consists of two actions: moving left or right.
Revised: The action space consists of two actions: moving left or right.

Original: We introduce two change factors for the state dynamics θ sk : varying gravity and varying mass of the cart, and a change factor in the observation function θ ok that is the image noise level.
Revised: We introduce two change factors for the state dynamics θ sk : varying gravity and varying mass of the cart, and a change factor in the observation function θ ok that is the image noise level.

Original: Fig.
Revised: Fig.

Original: A4 gives a visual example of Cartpole game, and the image with Gaussian noise.
Revised: A4 gives a visual example of Cartpole game, and the image with Gaussian noise.

Original: The images of the varying gravity and mass look exactly like the original image.
Revised: The images of the varying gravity and mass look exactly like the original image.

Original: Speciﬁcally, in the gravity case, we consider source domains with gravity g
Revised: Speciﬁcally, in the gravity case, we consider source domains with gravity g

Original: = { 5 , 10 , 20 , 30 , 40 } .
Revised: = { 5 , 10 , 20 , 30 , 40 } .

Original: We take into account both interpolation (where the gravity in the target domain is in the support of that in source domains) with g = { 15 } , and extrapolation (where it is out of the support w.r.t.
Revised: We take into account both interpolation (where the gravity in the target domain is in the support of that in source domains) with g = { 15 } , and extrapolation (where it is out of the support w.r.t.

Original: the source domains) with g = { 55 } .
Revised: the source domains) with g = { 55 } .

Original: Similarly, we consider source domains where the mass of the cart is m = { 0 . 5 , 1 . 5 , 2 . 5 , 3 . 5 , 4 . 5 } , while in target domains it is m = { 1 . 0 , 5 . 5 } .
Revised: Similarly, we consider source domains where the mass of the cart is m = { 0 . 5 , 1 . 5 , 2 . 5 , 3 . 5 , 4 . 5 } , while in target domains it is m = { 1 . 0 , 5 . 5 } .

Original: In terms of changes on the observation function θ o k , we add
Revised: In terms of changes on the observation function θ o k , we add

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: Gaussian noise on the images with variance σ = { 0 . 25 , 0 .
Revised: Gaussian noise on the images with variance σ = { 0 . 25 , 0 .

Original: in source domains, and σ = { 0 . 5 , 2 . 75 } in target domains.
Revised: in source domains, and σ = { 0 . 5 , 2 . 75 } in target domains.

Original: We summarize the detailed settings in both source and target domains in Table A1.
Revised: We summarize the detailed settings in both source and target domains in Table A1.

Original: In particular in each experiment we use all source domains for each change factor and one of the target domains at a time in either the interpolation and extrapolation set.
Revised: In particular in each experiment we use all source domains for each change factor and one of the target domains at a time in either the interpolation and extrapolation set.

Original: Gravity
Revised: Gravity

Original: Mass
Revised: Mass

Original: [Figure]
Revised: [Figure]

Original: Source domains
Revised: Source domains

Original: [Table]
Revised: [Table]

Original: Interpolation
Revised: Interpolation

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: A5.1. L EARNED θ k IN C ARTPOLE EXPERIMENTS
Revised: A5.1. L EARNED θ k IN C ARTPOLE EXPERIMENTS

Original: Fig.
Revised: Fig.

Original: A5 shows the estimated θ k in the modiﬁed Cartpole experiments.
Revised: A5 shows the estimated θ k in the modiﬁed Cartpole experiments.

Original: For the gravity and mass scenarios, the learned parameters with different sample sizes are close with each other.
Revised: For the gravity and mass scenarios, the learned parameters with different sample sizes are close with each other.

Original: This phenomenon indicates that even with only a few samples ( N target = 50) , AdaRL can estimate these change parameters very well.
Revised: This phenomenon indicates that even with only a few samples ( N target = 50) , AdaRL can estimate these change parameters very well.

Original: For the noise level factor, the learned curves with different sample sizes have a similar behaviour, but the distance is larger.
Revised: For the noise level factor, the learned curves with different sample sizes have a similar behaviour, but the distance is larger.

Original: We can see that the θ k we learn is approximately a linear function of the actual perturbation in gravity, while for the mass and noise are monotonic functions.
Revised: We can see that the θ k we learn is approximately a linear function of the actual perturbation in gravity, while for the mass and noise are monotonic functions.

Original: A5.1.
Revised: A5.1.

Original: A VERAGE FINAL SCORES FOR MULTIPLE N target IN C ARTPOLE EXPERIMENTS
Revised: A VERAGE FINAL SCORES FOR MULTIPLE N target IN C ARTPOLE EXPERIMENTS

Original: Tables A2 and A3 show the complete results of the modiﬁed Cartpole experiments (POMDP settings) for N target = 20 and N target = 10000 .
Revised: Tables A2 and A3 show the complete results of the modiﬁed Cartpole experiments (POMDP settings) for N target = 20 and N target = 10000 .

Original: Table A4 and A5 give the complete results of the modiﬁed
Revised: Table A4 and A5 give the complete results of the modiﬁed

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: Cartpole experiments (MDP settings with symbolic input) for N target = 20 and N target =
Revised: Cartpole experiments (MDP settings with symbolic input) for N target = 20 and N target =

Original: We average the scores across 30 trials from different random seeds during the policy learning stage.
Revised: We average the scores across 30 trials from different random seeds during the policy learning stage.

Original: The results suggest that, in most cases, AdaRL can outperform other baselines
Revised: The results suggest that, in most cases, AdaRL can outperform other baselines

Original: A5.1. A VERAGE POLICY LEARNING CURVES IN TERMS
Revised: A5.1. A VERAGE POLICY LEARNING CURVES IN TERMS

Original: STEPS
Revised: STEPS

Original: [Figure]
Revised: [Figure]

Original: Episodes °°
Revised: Episodes °°

Original: [Figure]
Revised: [Figure]

Original: Color: yellow
Revised: Color: yellow

Original: Oracle
Revised: Oracle

Original: AdaRL
Revised: AdaRL

Original: Non-t
Revised: Non-t

Original: PNN
Revised: PNN

Original: PSM
Revised: PSM

Original: MTQ
Revised: MTQ

Original: Episodes °°
Revised: Episodes °°

Original: [Figure]
Revised: [Figure]

Original: Color: yellow
Revised: Color: yellow

Original: Oracle
Revised: Oracle

Original: AdaRL
Revised: AdaRL

Original: Non-t
Revised: Non-t

Original: PNN
Revised: PNN

Original: PSM
Revised: PSM

Original: MTQ
Revised: MTQ

Original: Episodes °°
Revised: Episodes °°

Original: S c
Revised: S c

Original: o r e s
Revised: o r e s

Original: Color: yellow
Revised: Color: yellow

Original: Oracle
Revised: Oracle

Original: AdaRL
Revised: AdaRL

Original: [Figure]
Revised: [Figure]

Original: MTQ
Revised: MTQ

Original: [Figure]
Revised: [Figure]

Original: °
Revised: °

Original: [Figure]
Revised: [Figure]

Original: yellow
Revised: yellow

Original: [Figure]
Revised: [Figure]

Original: S c
Revised: S c

Original: o r e s
Revised: o r e s

Original: [Figure]
Revised: [Figure]

Original: Fig.
Revised: Fig.

Original: A7 and A6 provide the learning curves for modiﬁed Cartpole experiments (MDP and POMDP versions) with multiple change factors.
Revised: A7 and A6 provide the learning curves for modiﬁed Cartpole experiments (MDP and POMDP versions) with multiple change factors.

Original: In most cases, AdaRL can converge faster than other baselines
Revised: In most cases, AdaRL can converge faster than other baselines

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Color: yellow
Revised: Color: yellow

Original: [Figure]
Revised: [Figure]

Original: Color: yellow
Revised: Color: yellow

Original: [Figure]
Revised: [Figure]

Original: °°
Revised: °°

Original: [Figure]
Revised: [Figure]

Original: S c
Revised: S c

Original: o r e s
Revised: o r e s

Original: Color: yellow
Revised: Color: yellow

Original: [Figure]
Revised: [Figure]

Original: Color: yellow
Revised: Color: yellow

Original: [Figure]
Revised: [Figure]

Original: °°
Revised: °°

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Color: yellow
Revised: Color: yellow

Original: [Figure]
Revised: [Figure]

Original: Gravity:
Revised: Gravity:

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: A5. C OMPLETE RESULTS OF THE MODIFIED P ONG EXPERIMENT WITH CHANGING DYNAMICS AND OBSERVATIONS
Revised: A5. C OMPLETE RESULTS OF THE MODIFIED P ONG EXPERIMENT WITH CHANGING DYNAMICS AND OBSERVATIONS

Original: Atari Pong is a two-dimensional game that simulates table tennis.
Revised: Atari Pong is a two-dimensional game that simulates table tennis.

Original: The agent controls a paddle moving up and down vertically, aiming at hitting the ball.
Revised: The agent controls a paddle moving up and down vertically, aiming at hitting the ball.

Original: The goal for the agent is to reach higher scores, which are earned when the other agent (hard-coded) fails to hit back the ball.
Revised: The goal for the agent is to reach higher scores, which are earned when the other agent (hard-coded) fails to hit back the ball.

Original: We show the example of the original visual inputs and how it appears after we have changed each of the change factors in
Revised: We show the example of the original visual inputs and how it appears after we have changed each of the change factors in

Original: Fig.
Revised: Fig.

Original: A8.
Revised: A8.

Original: In source domains, the degrees are chosen from ω = { 0 ◦ , 180 ◦ } , and in target domains, they are chosen from ω = { 90 ◦ , 270 ◦ } .
Revised: In source domains, the degrees are chosen from ω = { 0 ◦ , 180 ◦ } , and in target domains, they are chosen from ω = { 90 ◦ , 270 ◦ } .

Original: For the image size, we reduce the original image by a factor of { 2 , 4 , 6 , 8 } in source domains and by a factor of { 3 , 9 } in target domains.
Revised: For the image size, we reduce the original image by a factor of { 2 , 4 , 6 , 8 } in source domains and by a factor of { 3 , 9 } in target domains.

Original: We summarize the detailed settings in both source and target domains in Table A6.
Revised: We summarize the detailed settings in both source and target domains in Table A6.

Original: In particular, in each experiment we use all source domains for each change factor and one of the target domains at a time in either the interpolation and extrapolation set.
Revised: In particular, in each experiment we use all source domains for each change factor and one of the target domains at a time in either the interpolation and extrapolation set.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]

Original: A5.2. L EARNED θ k IN MODIFIED P ONG EXPERIMENTS
Revised: A5.2. L EARNED θ k IN MODIFIED P ONG EXPERIMENTS

Original: Fig.
Revised: Fig.

Original: A9a, Table A and Table A8 show the learned θ k in modiﬁed Pong experiments across different change factors.
Revised: A9a, Table A and Table A8 show the learned θ k in modiﬁed Pong experiments across different change factors.

Original: [Table]
Revised: [Table]

Original: We can ﬁnd that each dimension of the learned θ sk is a nonlinear monotonic function of the change factors.
Revised: We can ﬁnd that each dimension of the learned θ sk is a nonlinear monotonic function of the change factors.

Original: Table A7, Table A8 and Fig.
Revised: Table A7, Table A8 and Fig.

Original: A9b also give the learned θ k with different sample sizes N target in target domains.
Revised: A9b also give the learned θ k with different sample sizes N target in target domains.

Original: Similarly, the learned curves with different sample sizes are homologous.
Revised: Similarly, the learned curves with different sample sizes are homologous.

Original: Even with a few samples, AdaRL can still capture the model changes well.
Revised: Even with a few samples, AdaRL can still capture the model changes well.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: A VERAGE FINAL SCORES FOR MULTIPLE N target
Revised: A VERAGE FINAL SCORES FOR MULTIPLE N target

Original: Table A2 and A3 provides the complete results of the modiﬁed Pong experiments with N target =and 10000 , respectively.
Revised: Table A2 and A3 provides the complete results of the modiﬁed Pong experiments with N target =and 10000 , respectively.

Original: The details of both source and target domains are listed in Table A6.
Revised: The details of both source and target domains are listed in Table A6.

Original: Similar to the results of Cartpole, AdaRL can perform the best among all baselines in Pong experiments.
Revised: Similar to the results of Cartpole, AdaRL can perform the best among all baselines in Pong experiments.

Original: As shown in the results of the main paper, AdaRL consistently outperforms the other methods.
Revised: As shown in the results of the main paper, AdaRL consistently outperforms the other methods.

Original: [Figure]
Revised: [Figure]

Original: A VERAGE POLICY LEARNING CURVES IN TERMS OF STEPS
Revised: A VERAGE POLICY LEARNING CURVES IN TERMS OF STEPS

Original: Fig.
Revised: Fig.

Original: A12 gives the learning curves for modiﬁed Pong experiments with multiple change factors.
Revised: A12 gives the learning curves for modiﬁed Pong experiments with multiple change factors.

Original: From the results, we can ﬁnd that AdaRL can converge faster than other baselines.
Revised: From the results, we can ﬁnd that AdaRL can converge faster than other baselines.

Original: A 
Revised: A

Original: [Figure]
Revised: [Figure]

Original:  C OMPLETE RESULTS OF THE MODIFIED P ONG EXPERIMENT WITH CHANGING REWARD FUNCTIONS
Revised:  C OMPLETE RESULTS OF THE MODIFIED P ONG EXPERIMENT WITH CHANGING REWARD FUNCTIONS

Original: Table A9 summarizes the detailed change factors in both linear and non-linear reward groups.
Revised: Table A9 summarizes the detailed change factors in both linear and non-linear reward groups.

Original: [Table]
Revised: [Table]

Original: We denote with L the half-length of the paddle and then formulate the two groups of reward functions as: (1) Linear reward functions: r t = k 1 dL , where k 1 ∈ { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 6 , 0 . 7 , 0 . 8 } in source domains and k 1 ∈ { 0 . 5 , 0 . 9 } in target domains; and (2) Non-linear reward functions: r t = k 2 L d +3 L , where k 2 ∈ { 2 . 0 , 3 . 0 , 5 . 0 , 6 . 0 , 7 . 0 , 8 . 0 , 9 . 0 } in source domains and k 2 ∈ { 1 . 0 , 4 . 0 } in target domains.
Revised: We denote with L the half-length of the paddle and then formulate the two groups of reward functions as: (1) Linear reward functions: r t = k 1 dL , where k 1 ∈ { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 6 , 0 . 7 , 0 . 8 } in source domains and k 1 ∈ { 0 . 5 , 0 . 9 } in target domains; and (2) Non-linear reward functions: r t = k 2 L d +3 L , where k 2 ∈ { 2 . 0 , 3 . 0 , 5 . 0 , 6 . 0 , 7 . 0 , 8 . 0 , 9 . 0 } in source domains and k 2 ∈ { 1 . 0 , 4 . 0 } in target domains.

Original: A5.3. L EARNED θ r IN MODIFIED P ONG EXPERIMENTS
Revised: A5.3. L EARNED θ r IN MODIFIED P ONG EXPERIMENTS

Original: Fig.
Revised: Fig.

Original: A10 and A11 give the learned θ r with both linear and non-linear rewards.
Revised: A10 and A11 give the learned θ r with both linear and non-linear rewards.

Original: In both groups, the learned θ r is linearly or monotonically correlated with the change factor k 1 and there is no signiﬁcant gap between the learned θ r with different N target .
Revised: In both groups, the learned θ r is linearly or monotonically correlated with the change factor k 1 and there is no signiﬁcant gap between the learned θ r with different N target .

Original: A5.3. A VERAGE FINAL SCORES FOR MULTIPLE N target
Revised: A5.3. A VERAGE FINAL SCORES FOR MULTIPLE N target

Original: Table A12, A13 and A14 shows the average ﬁnal scores for N target = 50 , 10000 and 50000 in modiﬁed Pong experiments with changing rewards.
Revised: Table A12, A13 and A14 shows the average ﬁnal scores for N target = 50 , 10000 and 50000 in modiﬁed Pong experiments with changing rewards.

Original: AdaRL consistently outperforms the other methods across different N target .
Revised: AdaRL consistently outperforms the other methods across different N target .

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: A VERAGE POLICY LEARNING CURVES IN TERMS OF STEPS
Revised: A VERAGE POLICY LEARNING CURVES IN TERMS OF STEPS

Original: Fig.
Revised: Fig.

Original: A12 (last two rows) gives the learning curves for modiﬁed Pong experiments with changing rewards.
Revised: A12 (last two rows) gives the learning curves for modiﬁed Pong experiments with changing rewards.

Original: [Figure]
Revised: [Figure]

Original: R ESULTS ON M U J O C O BENCHMARKS
Revised: R ESULTS ON M U J O C O BENCHMARKS

Original: We also apply AdaRL on MuJoCo benchmarks (Todorov et al., 2012), including Cheetah and Ant with a set of target velocities.
Revised: We also apply AdaRL on MuJoCo benchmarks (Todorov et al., 2012), including Cheetah and Ant with a set of target velocities.

Original: We follow the setup in MAML (Finn et al., 2017) and CAVIA (Zintgraf et al., 2019).
Revised: We follow the setup in MAML (Finn et al., 2017) and CAVIA (Zintgraf et al., 2019).

Original: In model estimation stage, we choose 20 tasks for each of the game.
Revised: In model estimation stage, we choose 20 tasks for each of the game.

Original: The goal velocity of each task is sampled between 0 . 0 and 2 . 0 for the cheetah and between 0 . 0 and 3 . 0 for ant.
Revised: The goal velocity of each task is sampled between 0 . 0 and 2 . 0 for the cheetah and between 0 . 0 and 3 . 0 for ant.

Original: The reward is the negative absolute value between agent’s current and the goal velocity.
Revised: The reward is the negative absolute value between agent’s current and the goal velocity.

Original: In policy optimization stage, we utilize Trust Region Policy Optimization (TRPO Schulman et al.
Revised: In policy optimization stage, we utilize Trust Region Policy Optimization (TRPO Schulman et al.

Original: The results on
Revised: The results on

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] 
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: Color: yellow
Revised: Color: yellow

Original: Oracle
Revised: Oracle

Original: AdaRL
Revised: AdaRL

Original: [Figure]
Revised: [Figure]

Original: Color: yellow
Revised: Color: yellow

Original: Oracle
Revised: Oracle

Original: [Figure]
Revised: [Figure]

Original: °°
Revised: °°

Original: [Figure]
Revised: [Figure]

Original: S c
Revised: S c

Original: o r e s
Revised: o r e s

Original: Color: yellow
Revised: Color: yellow

Original: [Figure]
Revised: [Figure]

Original: : yellow
Revised: : yellow

Original: [Figure]
Revised: [Figure]

Original: Color: yellow
Revised: Color: yellow

Original: [Figure]
Revised: [Figure]

Original: Color: yellow
Revised: Color: yellow

Original: [Figure]
Revised: [Figure]

Original: Color: yellow
Revised: Color: yellow

Original: [Figure]
Revised: [Figure]

Original: S c
Revised: S c

Original: o r e s
Revised: o r e s

Original: Color: yellow
Revised: Color: yellow

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: the target domains are speciﬁed in Table A15, suggesting that AdaRL can achieve better performance than the meta-learning approaches (i
Revised: the target domains are speciﬁed in Table A15, suggesting that AdaRL can achieve better performance than the meta-learning approaches (i

Original: [Table]
Revised: [Table]

Original: e., MAML and CAVIA)
Revised: e., MAML and CAVIA)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: A5. E FFECT OF THE POLICY USED FOR DATA COLLECTION DURING MODEL ESTIMATION
Revised: A5. E FFECT OF THE POLICY USED FOR DATA COLLECTION DURING MODEL ESTIMATION

Original: In our framework, we use the random policy to generate trajectories for each domain.
Revised: In our framework, we use the random policy to generate trajectories for each domain.

Original: The generated trajectories are further used to conduct the model estimation.
Revised: The generated trajectories are further used to conduct the model estimation.

Original: To study whether the random policy will affect the effectiveness of model estimation, we compare the learnt parameters of latent space in model estimation with the trajectories generated via (1) the random policy in our framework, and (2) the optimal policies learnt on the source domains.
Revised: To study whether the random policy will affect the effectiveness of model estimation, we compare the learnt parameters of latent space in model estimation with the trajectories generated via (1) the random policy in our framework, and (2) the optimal policies learnt on the source domains.

Original: Here we show the case with changing gravity in the modiﬁed Cartpole game.
Revised: Here we show the case with changing gravity in the modiﬁed Cartpole game.

Original: In Fig.
Revised: In Fig.

Original: A13, we give the learnt parameters ( µ and log σ ) of the ﬁrst and second components in the Gaussian mixtures of the 10 -th latent state at 20 -th epoch.
Revised: A13, we give the learnt parameters ( µ and log σ ) of the ﬁrst and second components in the Gaussian mixtures of the 10 -th latent state at 20 -th epoch.

Original: The results demonstrate that the difference between the two sets of learnt parameters is limited.
Revised: The results demonstrate that the difference between the two sets of learnt parameters is limited.

Original: A5. M ORE STATISTICAL EVALUATION PROTOCOLS ON THE PERFORMANCE
Revised: A5. M ORE STATISTICAL EVALUATION PROTOCOLS ON THE PERFORMANCE

Original: In this section, we provide a more detailed and comprehensive comparison on the performances of
Revised: In this section, we provide a more detailed and comprehensive comparison on the performances of

Original: AdaRL and other baseline methods.
Revised: AdaRL and other baseline methods.

Original: Following the recent published work on reliable evaluation for
Revised: Following the recent published work on reliable evaluation for

Original: RL (Agarwal et al., 2021b), we utilize 4 evaluation metrics: median, interquartile mean (IQM), mean, and optimality gap.
Revised: RL (Agarwal et al., 2021b), we utilize 4 evaluation metrics: median, interquartile mean (IQM), mean, and optimality gap.

Original: Median and mean are the sample median and mean, respectively.
Revised: Median and mean are the sample median and mean, respectively.

Original: IQM discards the top and bottom 25% samples and then computes the mean value of the remaining ones.
Revised: IQM discards the top and bottom 25% samples and then computes the mean value of the remaining ones.

Original: Thus, this factor is insensitive to outliers.
Revised: Thus, this factor is insensitive to outliers.

Original: Optimality gap is quantiﬁed via the gap between the performance of each method and the mean score obtained by the oracle agent.
Revised: Optimality gap is quantiﬁed via the gap between the performance of each method and the mean score obtained by the oracle agent.

Original: Therefore, higher mean, median and
Revised: Therefore, higher mean, median and

Original: IQM and lower optimality are the indications of better methods.
Revised: IQM and lower optimality are the indications of better methods.

Original: All metrics are with 95% bootstrap conﬁdence intervals (CIs)
Revised: All metrics are with 95% bootstrap conﬁdence intervals (CIs)

Original: (Efron, 1992).
Revised: (Efron, 1992).

Original: Fig.
Revised: Fig.

Original: A14, A15, A16, and A17 give the comparison on AdaRL and baselines using the set of evaluation protocols in different games and settings.
Revised: A14, A15, A16, and A17 give the comparison on AdaRL and baselines using the set of evaluation protocols in different games and settings.

Original: The results suggest that in most cases, AdaRL performs consistently better than the baselines across all evaluation metrics.
Revised: The results suggest that in most cases, AdaRL performs consistently better than the baselines across all evaluation metrics.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table][Table]
Revised: [Table]

Original: A E XPERIMENTAL DETAILS A6. H YPERPARAMETERS SELECTIONS
Revised: A E XPERIMENTAL DETAILS A6. H YPERPARAMETERS SELECTIONS

Original: Model estimation We use a random policy to collect sequence data from source domains.
Revised: Model estimation We use a random policy to collect sequence data from source domains.

Original: For both modiﬁed Cartpole and Pong experiments, the sequence length is 40 and the number of sequence is 10000 for each domain.
Revised: For both modiﬁed Cartpole and Pong experiments, the sequence length is 40 and the number of sequence is 10000 for each domain.

Original: The sampling resolution is set to be 0 .
Revised: The sampling resolution is set to be 0 .

Original: Other details are summarized in
Revised: Other details are summarized in

Original: Table A16.
Revised: Table A16.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Policy learning We adopt Double DQN (Van Hasselt et al., 2016) during policy learning stage.
Revised: Policy learning We adopt Double DQN (Van Hasselt et al., 2016) during policy learning stage.

Original: The detailed hyper-parameters are summarized in Table A17.
Revised: The detailed hyper-parameters are summarized in Table A17.

Original: For a fair comparison, we use the same set of hyperparameters for training other baseline methods.
Revised: For a fair comparison, we use the same set of hyperparameters for training other baseline methods.

Original: [Table]
Revised: [Table]

Original: E XPERIMENTAL PLATFORMS
Revised: E XPERIMENTAL PLATFORMS

Original: For the model estimation, Cartpole and Pong experiments are implemented on 1 NVIDIA P
Revised: For the model estimation, Cartpole and Pong experiments are implemented on 1 NVIDIA P

Original: GPUs and 4 NVidia V100 GPUs, respectively.
Revised: GPUs and 4 NVidia V100 GPUs, respectively.

Original: The policy learning stages in both experiments are implemented on 8 Nvidia RTX 1080Ti GPUs.
Revised: The policy learning stages in both experiments are implemented on 8 Nvidia RTX 1080Ti GPUs.

Original: A6. L ICENSES
Revised: A6. L ICENSES

Original: In our code, we have used the following libraries which are covered by the corresponding licenses:
Revised: In our code, we have used the following libraries which are covered by the corresponding licenses:

Original: • Tensorﬂow (Apache License 2.0), • Pytorch (BSD 3-Clause "New" or "Revised" License), • OpenAI Gym (MIT License), • OpenCV (Apache 2 License), • Numpy (BSD 3-Clause "New" or "Revised" License) • Keras (Apache License).
Revised: • Tensorﬂow (Apache License 2.0), • Pytorch (BSD 3-Clause "New" or "Revised" License), • OpenAI Gym (MIT License), • OpenCV (Apache 2 License), • Numpy (BSD 3-Clause "New" or "Revised" License) • Keras (Apache License).

Original: We plan to release our code under the MIT License.
Revised: We plan to release our code under the MIT License.