Original: Deep Reinforcement Learning amidst Lifelong Non-Stationarity
Revised: Deep Reinforcement Learning amidst Lifelong Non-Stationarity

Original: Anonymous Authors

Original: Abstract
Revised: Abstract

Original: As humans, our goals and our environment are persistently changing throughout our lifetime based on our experiences, actions, and internal and external drives.
Revised: As humans, our goals and our environment are persistently changing throughout our lifetime based on our experiences, actions, and internal and external drives.

Original: In contrast, typical reinforcement learning problem set-ups consider decision processes that are stationary across episodes.
Revised: In contrast, typical reinforcement learning problem set-ups consider decision processes that are stationary across episodes.

Original: Can we develop reinforcement learning algorithms that can cope with the persistent change in the former, more realistic problem settings?
Revised: Can we develop reinforcement learning algorithms that can cope with the persistent change in the former, more realistic problem settings?

Original: While on-policy algorithms such as policy gradients in principle can be extended to non-stationary settings, the same cannot be said for more efﬁcient off-policy algorithms that replay past experiences when learning.
Revised: While on-policy algorithms such as policy gradients in principle can be extended to non-stationary settings, the same cannot be said for more efﬁcient off-policy algorithms that replay past experiences when learning.

Original: In this work, we formalize this problem setting, and draw upon ideas from the online learning and probabilistic inference literature to derive an off-policy RL algorithm that can reason about and tackle such lifelong nonstationarity.
Revised: In this work, we formalize this problem setting, and draw upon ideas from the online learning and probabilistic inference literature to derive an off-policy RL algorithm that can reason about and tackle such lifelong nonstationarity.

Original: Our method leverages latent variable models to learn a representation of the environment from current and past experiences, and performs off-policy RL with this representation.
Revised: Our method leverages latent variable models to learn a representation of the environment from current and past experiences, and performs off-policy RL with this representation.

Original: We further introduce several simulation environments that exhibit lifelong non-stationarity, and empirically ﬁnd that our approach substantially outperforms approaches that do not reason about environment shift.
Revised: We further introduce several simulation environments that exhibit lifelong non-stationarity, and empirically ﬁnd that our approach substantially outperforms approaches that do not reason about environment shift.

Original: 1. Introduction
Revised: 1. Introduction

Original: In the standard reinforcement learning (RL) set-up, the agent is assumed to operate in a stationary environment with access to episodic resets.
Revised: In the standard reinforcement learning (RL) set-up, the agent is assumed to operate in a stationary environment with access to episodic resets.

Original: However, this assumption rarely holds in more realistic settings, especially in the context of lifelong learning systems (Thrun, 1998).
Revised: However, this assumption rarely holds in more realistic settings, especially in the context of lifelong learning systems (Thrun, 1998).

Original: That is, over the course of an agent’s lifetime, it may be subjected to environment dynamics and rewards that vary with time.In
Revised: That is, over the course of an agent’s lifetime, it may be subjected to environment dynamics and rewards that vary with time.

Original: Anonymous Institution, Anonymous City, Anonymous Region,

Original: Anonymous Country.

Original: to

Original: Under review International Conference ICML) Do robotics applications for example, this non-stationarity can manifest itself in changing terrains as the robot navigates to new, previously unexplored regions.
Revised: In robotics applications for example, this non-stationarity can manifest itself in changing terrains as the robot navigates to new, previously unexplored regions.

Original: In other situations, not even the objective is necessarily ﬁxed: consider an assistive robot helping a human whose preferences gradually change over time.
Revised: In other situations, not even the objective is necessarily ﬁxed: consider an assistive robot helping a human whose preferences gradually change over time.

Original: Since stationarity is a core assumption in many existing RL algorithms, they are unlikely to perform well in these environments.
Revised: Since stationarity is a core assumption in many existing RL algorithms, they are unlikely to perform well in these environments.

Original: Crucially, in each of the above scenarios, the environment is speciﬁed by unknown, time-varying parameters.
Revised: Crucially, in each of the above scenarios, the environment is speciﬁed by unknown, time-varying parameters.

Original: These latent parameters are also not i.i.d.
Revised: These latent parameters are also not i.i.d.

Original: , e.g., if the sky is clear at this very moment, it likely will not suddenly start raining in the next; in other words, these parameters have associated but unobserved dynamics.
Revised: , e.g., if the sky is clear at this very moment, it likely will not suddenly start raining in the next; in other words, these parameters have associated but unobserved dynamics.

Original: In this paper, we formalize this problem setting with the dynamic parameter Markov decision process (DP-MDP).
Revised: In this paper, we formalize this problem setting with the dynamic parameter Markov decision process (DP-MDP).

Original: The DP-MDP corresponds to a sequence of stationary MDPs, related through a set of latent parameters governed by an autonomous dynamical system.
Revised: The DP-MDP corresponds to a sequence of stationary MDPs, related through a set of latent parameters governed by an autonomous dynamical system.

Original: While all non-stationary MDPs are special instances of the partially observable Markov decision process (POMDP) (Kaelbling et al., 1998), in this setting, we can leverage structure available in the dynamics of the hidden parameters and avoid solving general POMDPs.
Revised: While all non-stationary MDPs are special instances of the partially observable Markov decision process (POMDP) (Kaelbling et al., 1998), in this setting, we can leverage structure available in the dynamics of the hidden parameters and avoid solving general POMDPs.

Original: On-policy RL algorithms can in principle cope with such non-stationarity (Sutton et al., 2007).
Revised: On-policy RL algorithms can in principle cope with such non-stationarity (Sutton et al., 2007).

Original: However, in highly dynamic environments, only a limited amount of interaction is permitted before the environment shifts, and on-policy methods may fail to adapt rapidly enough in this low-shot setting (Al-Shedivat et al., 2017).
Revised: However, in highly dynamic environments, only a limited amount of interaction is permitted before the environment shifts, and on-policy methods may fail to adapt rapidly enough in this low-shot setting (Al-Shedivat et al., 2017).

Original: Instead, we desire an off-policy RL algorithm that can use past experience both to improve sample efﬁciency and to reason about the environment dynamics.
Revised: Instead, we desire an off-policy RL algorithm that can use past experience both to improve sample efﬁciency and to reason about the environment dynamics.

Original: For the agent to adapt with the environment, it needs the ability to predict how the MDP parameters will shift.
Revised: For the agent to adapt with the environment, it needs the ability to predict how the MDP parameters will shift.

Original: We thus desire a representation of the MDP as well as a model of how the parameters evolve in this space, both of which can be learned from off-policy experience.
Revised: We thus desire a representation of the MDP as well as a model of how the parameters evolve in this space, both of which can be learned from off-policy experience.

Original: To this end, our core contribution is an off-policy RL algorithm that can operate under non-stationarity by jointly learning (1) a latent variable model, which lends a compact representation of the MDP, and (2) a maximum entropy policy with this representation.
Revised: To this end, our core contribution is an off-policy RL algorithm that can operate under non-stationarity by jointly learning (1) a latent variable model, which lends a compact representation of the MDP, and (2) a maximum entropy policy with this representation.

Original: We validate our approach, which we call Li felong L atent A ctor- C ritic (LILAC), on a set of simulated environments that demonstrate persis- tent non-stationarity, including a navigation task in an unbounded, non-episodic environment.
Revised: We validate our approach, which we call Li felong L atent A ctor- C ritic (LILAC), on a set of simulated environments that demonstrate persistent non-stationarity, including a navigation task in an un- bounded, non-episodic environment.

Original: In our experimental evaluation, we ﬁnd that our method far outperforms RL algorithms that do not account for environment dynamics.
Revised: In our experimental evaluation, we ﬁnd that our method far outperforms RL algorithms that do not account for environment dynamics.

Original: 2. Dynamic Parameter Markov Decision Processes
Revised: 2. Dynamic Parameter Markov Decision Processes

Original: The standard RL setting assumes episodic interaction with a ﬁxed MDP (Sutton & Barto, 2018).
Revised: The standard RL setting assumes episodic interaction with a ﬁxed MDP (Sutton & Barto, 2018).

Original: While this setting enables learning in highly structured environments, it is limited in expressivity due to the core assumption of fully observed,
Revised: While this setting enables learning in highly structured environments, it is limited in expressivity due to the core assumption of fully observed,

Original: Markovian dynamics.
Revised: Markovian dynamics.

Original: In the real world, the assumption of episodic interaction with identical MDPs is limiting as it does not capture the wide variety of exogenous factors that may effect the decision-making problem.
Revised: In the real world, the assumption of episodic interaction with identical MDPs is limiting as it does not capture the wide variety of exogenous factors that may effect the decision-making problem.

Original: A common model to avoid the strict assumption of Markovian observations is the partially observed MDP (POMDP) formulation
Revised: A common model to avoid the strict assumption of Markovian observations is the partially observed MDP (POMDP) formulation

Original: Kaelbling et al., 1998).
Revised: Kaelbling et al., 1998).

Original: While the POMDP is highly general, we focus in this work on leveraging known structure of the non-stationary MDP to improve performance.
Revised: While the POMDP is highly general, we focus in this work on leveraging known structure of the non-stationary MDP to improve performance.

Original: In particular, we consider an episodic environment, which we call the dynamic parameter MDP (DP-MDP), where a new MDP (we also refer to MDPs as tasks) is presented in each episode.
Revised: In particular, we consider an episodic environment, which we call the dynamic parameter MDP (DP-MDP), where a new MDP (we also refer to MDPs as tasks) is presented in each episode.

Original: In reﬂection of the regularity of real-world non-stationarity, the tasks are sequentially related through a set of continuous parameters.
Revised: In reﬂection of the regularity of real-world non-stationarity, the tasks are sequentially related through a set of continuous parameters.

Original: Formally, the DP-MDP is equipped with state space S , action space A , and initial state distribution ρ s ( s 1 ) .
Revised: Formally, the DP-MDP is equipped with state space S , action space A , and initial state distribution ρ s ( s 1 ) .

Original: Following the formulation of the Hidden Parameter MDP (HiP-MDP) (Doshi-Velez & Konidaris, 2016), a set of unobserved task parameters z ∈ Z deﬁnes the dynamics p s ( s t +1 | s t , a t ; z ) and reward function r ( s t , a t ; z ) for each task.
Revised: Following the formulation of the Hidden Parameter MDP (HiP-MDP) (Doshi-Velez & Konidaris, 2016), a set of unobserved task parameters z ∈ Z deﬁnes the dynamics p s ( s t +1 | s t , a t ; z ) and reward function r ( s t , a t ; z ) for each task.

Original: In contrast to the HiP-MDP, the task parameters z in the DP-MDP are not sampled i.i.d.
Revised: In contrast to the HiP-MDP, the task parameters z in the DP-MDP are not sampled i.i.d.

Original: but instead shift stochastically according to p z ( z i +1 |
Revised: but instead shift stochastically according to p z ( z i +1 |

Original: z i ) , with initial distribution ρ z ( z 1 ) .
Revised: z i ) , with initial distribution ρ z ( z 1 ) .

Original: In other words, the DP-MDP is a sequence of tasks with parameters determined by the transition function p z .
Revised: In other words, the DP-MDP is a sequence of tasks with parameters determined by the transition function p z .

Original: The DP-MDP can be also viewed as a hidden Markov model wherein the state represents the MDP in each episode and observations of the hidden Markov model are given in the form of trajectories collected by the agent.
Revised: The DP-MDP can be also viewed as a hidden Markov model wherein the state represents the MDP in each episode and observations of the hidden Markov model are given in the form of trajectories collected by the agent.

Original: If the task parameters z for each episode were known, the augmented state space S
Revised: If the task parameters z for each episode were known, the augmented state space S

Original: × Z would deﬁne a fully observable MDP for which we can use standard RL algorithms.
Revised: × Z would deﬁne a fully observable MDP for which we can use standard RL algorithms.

Original: Hence, in our approach, we aim to infer the hidden task parameters and learn their transition function, allowing us to leverage existing RL algorithms by augmenting the observations with the inferred task parameters.
Revised: Hence, in our approach, we aim to infer the hidden task parameters and learn their transition function, allowing us to leverage existing RL algorithms by augmenting the observations with the inferred task parameters.

Original: 3. Preliminaries: RL as Inference
Revised: 3. Preliminaries: RL as Inference

Original: We ﬁrst discuss an established connection between probabilistic inference and reinforcement learning (Toussaint, 2009; Levine, 2018) to provide some context for our approach.
Revised: We ﬁrst discuss an established connection between probabilistic inference and reinforcement learning (Toussaint, 2009; Levine, 2018) to provide some context for our approach.

Original: At a high level, this framework casts sequential decision-making as a probabilistic graphical model, and from this perspective, the maximum-entropy RL objective can be derived as an inference procedure in this model.
Revised: At a high level, this framework casts sequential decision-making as a probabilistic graphical model, and from this perspective, the maximum-entropy RL objective can be derived as an inference procedure in this model.

Original: 3.1. A Probabilistic Graphical Model for RL
Revised: 3.1. A Probabilistic Graphical Model for RL

Original: As depicted in Figure 1, the proposed model consists of states s t , actions a t , and per-timestep optimality variables
Revised: As depicted in Figure 1, the proposed model consists of states s t , actions a t , and per-timestep optimality variables

Original: O t , which are related to rewards by p ( O t = 1 | s t , a t ) =
Revised: O t , which are related to rewards by p ( O t = 1 | s t , a t ) =

Original: exp( r ( s t , a t )) and denote whether the action a t taken from state s t is optimal.
Revised: exp( r ( s t , a t )) and denote whether the action a t taken from state s t is optimal.

Original: While rewards are required to be non-positive through this relation, so long the rewards are bounded, they can be scaled and centered to be no greater than 0.
Revised: While rewards are required to be non-positive through this relation, so long the rewards are bounded, they can be scaled and centered to be no greater than 0.

Original: A trajectory is the sequence of states and actions, ( s 1 , a 1 , s 2 , . . .
Revised: A trajectory is the sequence of states and actions, ( s 1 , a 1 , s 2 , . . .

Original: , s T , a T ) , and we aim to infer the posterior distribution p ( s 1: T , a 1: T |O 1: T = 1) , i.e., the trajectory distribution that is optimal for all timesteps.
Revised: , s T , a T ) , and we aim to infer the posterior distribution p ( s 1: T , a 1: T |O 1: T = 1) , i.e., the trajectory distribution that is optimal for all timesteps.

Original: 3.2. Variational Inference
Revised: 3.2. Variational Inference

Original: Among existing inference tools, structured variational inference is particularly appealing for its scalability and efﬁciency to approximate the distribution of interest.
Revised: Among existing inference tools, structured variational inference is particularly appealing for its scalability and efﬁciency to approximate the distribution of interest.

Original: In the variational inference framework, a variational distribution q is optimized through the variational lower bound to approximate another distribution p .
Revised: In the variational inference framework, a variational distribution q is optimized through the variational lower bound to approximate another distribution p .

Original: Assuming a uniform prior over actions, the optimal trajectory distribution is:
Revised: Assuming a uniform prior over actions, the optimal trajectory distribution is:

Original: [Equation]
Revised: [Equation]

Original: exp

Original: [Equation]

Original: a

Original: [Equation]

Original: | s t [Equation] a t ) . For the approximating distribution q , we choose the form q ( s 1: T , a 1: T ) =
Revised: For the approximating distribution q , we choose the form q ( s 1: T , a 1: T ) =

Original: p ( s 1 ) (cid:81) Tt =1 p ( s t +1 | s t , a t ) q ( a t
Revised: p ( s 1 ) (cid:81) Tt =1 p ( s t +1 | s t , a t ) q ( a t

Original: | s t ) , where p ( s 1 ) and p ( s t +1 | s t , a t ) are ﬁxed and given by the environment.
Revised: | s t ) , where p ( s 1 ) and p ( s t +1 | s t , a t ) are ﬁxed and given by the environment.

Original: We now rename q ( a t | s t ) to π ( a t | s t ) since this represents the desired policy.
Revised: We now rename q ( a t | s t ) to π ( a t | s t ) since this represents the desired policy.

Original: By Jensen’s inequality, the variational lower bound for the evidence O 1: T = 1 is log p ( O 1: T = 1) =
Revised: By Jensen’s inequality, the variational lower bound for the evidence O 1: T = 1 is log p ( O 1: T = 1) =

Original: log E q (cid:20) p
Revised: log E q (cid:20) p

Original: ( s 1: T , a 1: T , O 1: T = 1) q ( s 1: T , a 1: T ) (cid:21) 
Revised: ( s 1: T , a 1: T , O 1: T = 1) q ( s 1: T , a 1: T ) (cid:21) [Equation] (cid:34)

Original: [Equation]
Revised: [Equation]

Revised: (cid:88) t

Original:  , which is the maximum entropy RL objective (Ziebart et al., 2008; Toussaint, 2009; Rawlik et al., 2013; Fox et al., 2015;
Revised: = r ( s t , a [Equation] ) − log π ( a t | s t ) (cid:35) , which is the maximum entropy RL objective (Ziebart et al., 2008; Toussaint, 2009; Rawlik et al., 2013; Fox et al., 2015;

Original: Haarnoja et al., 2017).
Revised: Haarnoja et al., 2017).

Original: This objective adds a conditional entropy term and thus maximizes both returns and the entropy of the policy.
Revised: This objective adds a conditional entropy term and thus maximizes both returns and the entropy of the policy.

Original: This formulation is known for its improvements in exploration, robustness, and stability over other RL algorithms, thus we build upon it in our method to inherit these qualities.
Revised: This formulation is known for its improvements in exploration, robustness, and stability over other RL algorithms, thus we build upon it in our method to inherit these qualities.

Original: We capture non-stationarity by augmenting the RL-as-inference model with latent variables
Revised: We capture non-stationarity by augmenting the RL-as-inference model with latent variables

Original: z i for each task i .
Revised: z i for each task i .

Original: As we will see in the next section, by viewing non-stationarity from this probabilistic perspective, our algorithm can be derived as an inference procedure in a uniﬁed model.
Revised: As we will see in the next section, by viewing non-stationarity from this probabilistic perspective, our algorithm can be derived as an inference procedure in a uniﬁed model.

Original: 4. Off-Policy Reinforcement Learning in Non-Stationary Environments
Revised: 4. Off-Policy Reinforcement Learning in Non-Stationary Environments

Original: Building upon the RL-as-inference framework, in this section, we offer a probabilistic graphical model that underlies the dynamic parameter MDP setting introduced in Section 2.
Revised: Building upon the RL-as-inference framework, in this section, we offer a probabilistic graphical model that underlies the dynamic parameter MDP setting introduced in Section 2.

Original: Then, using tools from variational inference, we derive a variational lower bound that performs joint RL and representation learning.
Revised: Then, using tools from variational inference, we derive a variational lower bound that performs joint RL and representation learning.

Original: Finally, we present our RL algorithm, which we call Li felong L atent A ctor- C ritic (LILAC), that optimizes this objective and builds upon on soft actorcritic (Haarnoja et al., 2018), an off-policy maximum entropy RL algorithm.
Revised: Finally, we present our RL algorithm, which we call Li felong L atent A ctor- C ritic (LILAC), that optimizes this objective and builds upon on soft actorcritic (Haarnoja et al., 2018), an off-policy maximum entropy RL algorithm.

Original: 4.1. Non-stationarity as a Probabilistic Model
Revised: 4.1. Non-stationarity as a Probabilistic Model

Original: We can cast the dynamic parameter MDP as a probabilistic hierarchical model, where non-stationarity occurs at the episodic level, and within each episode is an instance of a stationary MDP.
Revised: We can cast the dynamic parameter MDP as a probabilistic hierarchical model, where non-stationarity occurs at the episodic level, and within each episode is an instance of a stationary MDP.

Original: To do so, we construct a two-tiered model: on the ﬁrst level, we have the sequence of latent variables z i as a Markov chain, and on the second level, a Markov decision process corresponding to each z i .
Revised: To do so, we construct a two-tiered model: on the ﬁrst level, we have the sequence of latent variables z i as a Markov chain, and on the second level, a Markov decision process corresponding to each z i .

Original: The graphical model formulation of the DP-MDP is illustrated in Figure 2.
Revised: The graphical model formulation of the DP-MDP is illustrated in Figure 2.

Original: Within this formulation, the trajectories gathered from each episode are modeled individually, rather than amortized as in Subsection 3.2, and the joint probability distribution is deﬁned as follows:
Revised: Within this formulation, the trajectories gathered from each episode are modeled individually, rather than amortized as in Subsection 3.2, and the joint probability distribution is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where the probability of each trajectory τ given z , assuming a uniform prior over actions, isp
Revised: where the probability of each trajectory τ given z , assuming a uniform prior over actions, is

Original: [Equation]
Revised: [Equation]

Original: With this factorization, the non-stationary elements of the environment are captured by the latent variables z , and within a task, the dynamics and reward functions are necessarily stationary.
Revised: With this factorization, the non-stationary elements of the environment are captured by the latent variables z , and within a task, the dynamics and reward functions are necessarily stationary.

Original: This suggests that learning to infer z , which amounts to representing the non-stationarity elements of the environment with z , will reduce this RL setting to a stationary one.
Revised: This suggests that learning to infer z , which amounts to representing the non-stationarity elements of the environment with z , will reduce this RL setting to a stationary one.

Original: Taking this type of approach is appealing since there already exists a rich body of algorithms for the standard RL setting.
Revised: Taking this type of approach is appealing since there already exists a rich body of algorithms for the standard RL setting.

Original: In the next subsection, we describe how we can approximate the posterior over z , by deriving the evidence lower bound for this model under the variational inference framework.
Revised: In the next subsection, we describe how we can approximate the posterior over z , by deriving the evidence lower bound for this model under the variational inference framework.

Original: 4.2. Joint Representation and Reinforcement Learning via Variational Inference
Revised: 4.2. Joint Representation and Reinforcement Learning via Variational Inference

Original: Recall the agent is operating in an online learning setting.
Revised: Recall the agent is operating in an online learning setting.

Original: That is, it must continuously adapt to a stream of tasks and leverage experience gathered from previous tasks for learning.
Revised: That is, it must continuously adapt to a stream of tasks and leverage experience gathered from previous tasks for learning.

Original: Thus, at any episode i > 1 , the agent has observed all of the trajectories collected from episodes 1 through i − 1 , τ 1: i − 1 = { τ 1 , · · · , τ i − 1 } , where τ = { s 1 , a 1 , r 1 , . . .
Revised: Thus, at any episode i > 1 , the agent has observed all of the trajectories collected from episodes 1 through i − 1 , τ 1: i − 1 = { τ 1 , · · · , τ i − 1 } , where τ = { s 1 , a 1 , r 1 , . . .

Original: , s T , a T , r T } .
Revised: , s T , a T , r T } .

Original: We aim to infer, at every episode i , the posterior distribution over actions, given the evidence O i 1: T = 1 and the experience from the previous episodes τ 1: i − 1 .
Revised: We aim to infer, at every episode i , the posterior distribution over actions, given the evidence O i 1: T = 1 and the experience from the previous episodes τ 1: i − 1 .

Original: Following
Revised: Following

Original: Subsection 3.2, we can leverage variational inference to optimize a variational lower bound to the log-probability of this set of evidence, log p ( τ 1: i − 1 , O i 1: T = 1) .
Revised: Subsection 3.2, we can leverage variational inference to optimize a variational lower bound to the log-probability of this set of evidence, log p ( τ 1: i − 1 , O i 1: T = 1) .

Original: Since p ( τ 1: i − 1 , O i 1: T = 1) factorizes as p ( τ 1: i − 1 ) p
Revised: Since p ( τ 1: i − 1 , O i 1: T = 1) factorizes as p ( τ 1: i − 1 ) p

Original: ( O i 1: T = 1 | τ 1: i − 1 ) , the log-probability of the evidence can be decomposed into log p ( τ 1: i − 1 ) + log p ( O i 1: T = 1 | τ 1: i − 1 ) .
Revised: ( O i 1: T = 1 | τ 1: i − 1 ) , the log-probability of the evidence can be decom- posed into log p ( τ 1: i − 1 ) + log p ( O i 1: T = 1 | τ 1: i − 1 ) .

Original: These two terms can be separately lower bounded and summed to form a single objective.
Revised: These two terms can be separately lower bounded and summed to form a single objective.

Original: The variational lower bound of the ﬁrst term follows from that of a variational auto-encoder (Kingma & Welling, 2014) with evidence τ 1: i − 1 and latent variables z 1: i − 1 : log
Revised: The variational lower bound of the ﬁrst term follows from that of a variational auto-encoder (Kingma & Welling, 2014) with evidence τ 1: i − 1 and latent variables z 1: i − 1 : log

Original: [Equation]
Revised: [Equation]

Original: We choose our approximating distribution over the latent variables
Revised: We choose our approximating distribution over the latent variables

Original: z i to be conditioned on the trajectory from episode i , i.e. q ( z i | τ i ) .
Revised: z i to be conditioned on the trajectory from episode i , i.e. q ( z i | τ i ) .

Original: Then, the variational lower bound can be expressed as:
Revised: Then, the variational lower bound can be expressed as:

Original: [Equation]
Revised: [Equation]

Original: The lower bound L rep corresponds to an objective for unsupervised representation learning in a sequential latent variable model.
Revised: The lower bound L rep corresponds to an objective for unsupervised representation learning in a sequential latent variable model.

Original: By optimizing the reconstruction loss of the transitions and rewards for each episode, the learned latent variables should encode the varying parameters of the MDP.
Revised: By optimizing the reconstruction loss of the transitions and rewards for each episode, the learned latent variables should encode the varying parameters of the MDP.

Original: Further, by imposing the prior p ( z i |
Revised: Further, by imposing the prior p ( z i |

Original: z
Revised: z

Original: i − 1 ) on the approximated distribution q through the KL divergence, the latent variables are encouraged to be sequentially consistent across time.
Revised: i − 1 ) on the approximated distribution q through the KL divergence, the latent variables are encouraged to be sequentially consistent across time.

Original: This prior corresponds to a model of the environment’s latent dynamics and gives the agent a predictive estimate of future conditions of the environment (to the extent to which the DP-MDP is predictable).
Revised: This prior corresponds to a model of the environment’s latent dynamics and gives the agent a predictive estimate of future conditions of the environment (to the extent to which the DP-MDP is predictable).

Original: For the second term,
Revised: For the second term,

Original: [Equation]
Revised: [Equation]

Original: The ﬁnal inequality is given by steps from Subsection 3.2.
Revised: The ﬁnal inequality is given by steps from Subsection 3.2.

Original: The bound L RL optimizes for both policy returns and policy entropy, as in the maximum entropy RL objective, but here the policy is also conditioned on the inferred latent embeddings of the MDP.
Revised: The bound L RL optimizes for both policy returns and policy entropy, as in the maximum entropy RL objective, but here the policy is also conditioned on the inferred latent embeddings of the MDP.

Original: This objective essentially performs task-conditioned reinforcement learning where the task variables at episode i are given by p ( z i | τ 1: i − 1 ) .
Revised: This objective essentially performs task-conditioned reinforcement learning where the task variables at episode i are given by p ( z i | τ 1: i − 1 ) .

Original: Learning a multi-task RL policy is appealing, especially over a policy that adapts between episodes.
Revised: Learning a multi-task RL policy is appealing, especially over a policy that adapts between episodes.

Original: That is, if the shifts in the environment are similar to those seen previously, we do not expect its performance to degrade even if the environment is shifting quickly, whereas a single-task policy would likely struggle to adapt quickly enough.
Revised: That is, if the shifts in the environment are similar to those seen previously, we do not expect its performance to degrade even if the environment is shifting quickly, whereas a single-task policy would likely struggle to adapt quickly enough.

Original: Our proposed objective is the sum of the above two terms
Revised: Our proposed objective is the sum of the above two terms

Original: L = L rep + L RL , which is also a variational lower bound for our entire model.
Revised: L = L rep + L RL , which is also a variational lower bound for our entire model.

Original: Hence, while our objective was derived from and can be understood as an inference procedure in our probabilistic model, it also decomposes into two very intuitive objectives, with the ﬁrst corresponding to unsupervised representation learning and the second corresponding to reinforcement learning.
Revised: Hence, while our objective was derived from and can be understood as an inference procedure in our probabilistic model, it also decomposes into two very intuitive objectives, with the ﬁrst corresponding to unsupervised representation learning and the second corresponding to reinforcement learning.

Original: 4.3. Implementation Details
Revised: 4.3. Implementation Details

Original: To optimize the above objective, we extend soft actor-critic (SAC) (Haarnoja et al., 2018), which implements maximum entropy off-policy RL.
Revised: To optimize the above objective, we extend soft actor-critic (SAC) (Haarnoja et al., 2018), which implements maximum entropy off-policy RL.

Original: We introduce an inference network that outputs a distribution over latent variables for the i -th episode, q ( z i | τ i ) , conditioned on the trajectory from the i -th episode.
Revised: We introduce an inference network that outputs a distribution over latent variables for the i -th episode, q ( z i | τ i ) , conditioned on the trajectory from the i -th episode.

Original: The inference network, parameterized as a feedforward neural network, outputs parameters of a Gaussian distribution, and we use the reparameterization trick (Kingma & Welling, 2014) to sample z .
Revised: The inference network, parameterized as a feedforward neural network, outputs parameters of a Gaussian distribution, and we use the reparameterization trick (Kingma & Welling, 2014) to sample z .

Original: A decoder neural network reconstructs transitions and rewards given the latent embedding z i , current state s t , and action taken a t , i.e. p ( s t +1 | s t , a t ; z i ) and p ( r t | s t , a t ; z i ) .
Revised: A decoder neural network reconstructs transitions and rewards given the latent embedding z i , current state s t , and action taken a t , i.e. p ( s t +1 | s t , a t ; z i ) and p ( r t | s t , a t ; z i ) .

Original: Finally, p ( z i |
Revised: Finally, p ( z i |

Original: z
Revised: z

Original: i − 1 ) and p ( z i | τ 1: i − 1 ) are approximated with a shared long short-term memory (LSTM) network (Hochreiter &
Revised: i − 1 ) and p ( z i | τ 1: i − 1 ) are approximated with a shared long short-term memory (LSTM) network (Hochreiter &

Original: Schmidhuber, 1997), which, at each episode i , receives z i − from q ( z i − 1 | τ i − 1 ) and hidden state h
Revised: Schmidhuber, 1997), which, at each episode i , receives z i −from q ( z i − 1 | τ i − 1 ) and hidden state h

Original: i − 1 , and produces z i and the next hidden state
Revised: i − 1 , and produces z i and the next hidden state

Original: h
Revised: h

Original: i .
Revised: i .

Original: We visualize the entire network at a high level and how the different components interact in Figure 3.
Revised: We visualize the entire network at a high level and how the different components interact in Figure 3.

Original: As depicted, the policy and critic are both conditioned on the environment state and the latent variables z .
Revised: As depicted, the policy and critic are both conditioned on the environment state and the latent variables z .

Original: During training, z is sampled from q ( z i | τ i ) outputted by the inference network.
Revised: During training, z is sampled from q ( z i | τ i ) outputted by the inference network.

Original: At execution time, the latent variables z the policy receives are given by the LSTM network, based on the inferred latent variables from the previous episode.
Revised: At execu- tion time, the latent variables z the policy receives are given by the LSTM network, based on the inferred latent variables from the previous episode.

Original: Following SAC (Haarnoja et al., 2018), the actor loss J π and critic loss J Q areJ π =
Revised: Following SAC (Haarnoja et al., 2018), the actor loss J π and critic loss J Q are

Original: [Equation]
Revised: [Equation]

Original: where V denotes the target network.
Revised: where V denotes the target network.

Original: Our complete algorithm, Lifelong Latent Actor-Critic (LILAC), is summarized in Algorithm 1. Algorithm 
Revised: Our complete algorithm, Lifelong Latent Actor-Critic (LILAC), is summarized in Algorithm 1. Algorithm 1 Lifelong Latent Actor-Critic (LILAC)

Original: Input: env, α Q , α π , α φ , α dec , α p
Revised: Input: env, α Q , α π , α φ , α dec , α p

Original: Randomly initialize θ Q , θ π , φ , θ dec , and θ p
Revised: Randomly initialize θ Q , θ π , φ , θ dec , and θ p

Original: Initialize empty replay buffer D
Revised: Initialize empty replay buffer D

Original: Assign z 0 ← (cid:126)for i = 1, 2, ... do
Revised: Assign z 0 ← (cid:126)for i = 1, 2, ... do

Original: Sample z i ∼ p ψ ( z i
Revised: Sample z i ∼ p ψ ( z i

Original: | z i − 1 )
Revised: | z i − 1 )

Original: Collect trajectory τ i with π θ ( a | s , z )
Revised: Collect trajectory τ i with π θ ( a | s , z )

Original: Update replay buffer D
Revised: Update replay buffer D

Original: [ i ] ← τ i for j = 1, 2, ..., N do
Revised: [ i ] ← τ i for j = 1, 2, ..., N do

Original: Sample a batch of episodes E from D (cid:46) Update actor and critic θ Q ← θ Q − α Q ∇ θ Q J Q θ π ← θ π − α π ∇ θ π J π (cid:46) Update inference network φ ← φ − α φ ∇ φ ( J dec + J KL + J Q ) (cid:46) Update model θ dec ← θ dec − α dec ∇ θ dec J dec θ p ← θ p − α p ∇θ p J KL end for end for
Revised: Sample a batch of episodes E from D (cid:46) Update actor and critic θ Q ← θ Q − α Q ∇ θ Q J Q θ π ← θ π − α π ∇ θ π J π (cid:46) Update inference network φ ← φ − α φ ∇ φ ( J dec + J KL + J Q ) (cid:46) Update model θ dec ← θ dec − α dec ∇ θ dec J dec θ p ← θ p − α p ∇

Original: 5. Related Work
Revised: 5. Related Work

Original: While the POMDP formulation can capture non-stationarity and partial observability in sequential decision-making problems, exact solution methods are tractable only for tiny state and actions spaces (Kaelbling et al., 1998).
Revised: While the POMDP formulation can capture non-stationarity and partial observability in sequential decision-making problems, exact solution methods are tractable only for tiny state and actions spaces (Kaelbling et al., 1998).

Original: Representation learning, and especially deep learning paired with amortized variational inference, has enabled scaling of the POMDP formulation to a larger class of problems, including continuous state and action spaces (Igl et al., 2018; Han et al., 2020; Hafner et al., 2019) and image observations (Lee et al., 2019a;
Revised: Representation learning, and especially deep learning paired with amortized variational inference, has enabled scaling of the POMDP formulation to a larger class of problems, including continuous state and action spaces (Igl et al., 2018; Han et al., 2020; Hafner et al., 2019) and image observations (Lee et al., 2019a;

Original: Kapturowski et al., 2019).
Revised: Kapturowski et al., 2019).

Original: However, the generality of the POMDP ignores performance improvements that may be realized by exploiting the structure of the DP-MDP problem, and does not explicitly consider between-episode non-stationarity.
Revised: However, the generality of the POMDP ignores performance improvements that may be realized by exploiting the structure of the DP-MDP problem, and does not explicitly consider between-episode non-stationarity.

Original: A variety of intermediate problem statements between episodic MDPs and POMDPs have been proposed.
Revised: A variety of intermediate problem statements between episodic MDPs and POMDPs have been proposed.

Original: The
Revised: The

Original: Bayes-adaptive MDP formulation (BAMDP) (Duff, 2002;
Revised: Bayes-adaptive MDP formulation (BAMDP) (Duff, 2002;

Original: Ross et al., 2008), as well as the closely related hidden parameter MDP (HiP-MDP) (Doshi-Velez & Konidaris, 2016) consider an MDP with unknown parameters governing the reward and dynamics, which we aim to infer online over the course of one episode.
Revised: Ross et al., 2008), as well as the closely related hidden parameter MDP (HiP-MDP) (Doshi-Velez & Konidaris, 2016) consider an MDP with unknown parameters governing the reward and dynamics, which we aim to infer online over the course of one episode.

Original: In this formulation, the explorationexploitation dilemma is resolved by augmenting the state space with a representation of posterior belief over the latent parameters.
Revised: In this formulation, the explorationexploitation dilemma is resolved by augmenting the state space with a representation of posterior belief over the latent parameters.

Original: As noted by Duff (2002) in the RL literature and Feldbaum (1960); Bar-Shalom & Tse (1974) in control theory, this representation rapidly becomes intractable due to exploding state dimensionality.
Revised: As noted by Duff (2002) in the RL literature and Feldbaum (1960); Bar-Shalom & Tse (1974) in control theory, this representation rapidly becomes intractable due to exploding state dimensionality.

Original: As with the POMDP setting, recent work has developed effective methods for policy optimization in BAMDPs via, primarily, amortized inference (Zintgraf et al., 2020;
Revised: As with the POMDP setting, recent work has developed effective methods for policy optimization in BAMDPs via, primarily, amortized inference (Zintgraf et al., 2020;

Original: Rakelly et al., 2019; Lee et al., 2019b).
Revised: Rakelly et al., 2019; Lee et al., 2019b).

Original: However, the BAMDP framework does not address the dynamics of the latent parameter between episodes, assuming a temporally-ﬁxed structure.
Revised: However, the BAMDP framework does not address the dynamics of the latent parameter between episodes, assuming a temporally-ﬁxed structure.

Original: In contrast to the BAMDP formulation, we are capable of modeling the evolution of the latent variable over the course of episodes, leading to better priors for online inference.
Revised: In contrast to the BAMDP formulation, we are capable of modeling the evolution of the latent variable over the course of episodes, leading to better priors for online inference.

Original: A strongly related setting to the DP-MDP is the hiddenmode
Revised: A strongly related setting to the DP-MDP is the hiddenmode

Original: MDP (Choi et al., 2000).
Revised: MDP (Choi et al., 2000).

Original: The HM-MDP proposes to augment an MDP with a latent parameter that evolves via a hidden Markov model with a discrete number of states.
Revised: The HM-MDP proposes to augment an MDP with a latent parameter that evolves via a hidden Markov model with a discrete number of states.

Original: In both the HM-MDP and the DP-MDP, the latent variable evolves infrequently, as opposed to at every time step as in a
Revised: In both the HM-MDP and the DP-MDP, the latent variable evolves infrequently, as opposed to at every time step as in a

Original: POMDP setting.
Revised: POMDP setting.

Original: While the HM-MDP does not connect the non-stationarity of the latent parameter to the episodic RL problem, it is limited to a ﬁxed number of latent variable states due to the use of standard HMM inference algorithms.
Revised: While the HM-MDP does not connect the non-stationarity of the latent parameter to the episodic RL problem, it is limited to a ﬁxed number of latent variable states due to the use of standard HMM inference algorithms.

Original: In contrast, our approach allows continuous latent variables, thus widely extending the range of applicability.
Revised: In contrast, our approach allows continuous latent variables, thus widely extending the range of applicability.

Original: Non-stationarity in learning .
Revised: Non-stationarity in learning .

Original: LILAC shares conceptual similarities with methods from online learning and lifelong learning (Shalev-Shwartz, 2012; Gama et al., 2014), which aim to capture non-stationarity in supervised learning, as well as meta-learning and meta-reinforcement learning algorithms, which aim to rapidly adapt to new settings.
Revised: LILAC shares conceptual similarities with methods from online learning and lifelong learning (Shalev-Shwartz, 2012; Gama et al., 2014), which aim to capture non-stationarity in supervised learning, as well as meta-learning and meta-reinforcement learning algorithms, which aim to rapidly adapt to new settings.

Original: Meta-learning (Schmidhuber, 1987) algorithms learn an efﬁcient adaptation procedure via meta-training on a variety of tasks, such that learning in a new task can be performed as efﬁciently as possible (Finn et al., 2017).
Revised: Meta-learning (Schmidhuber, 1987) algorithms learn an efﬁcient adaptation procedure via meta-training on a variety of tasks, such that learning in a new task can be performed as efﬁciently as possible (Finn et al., 2017).

Original: Within meta-reinforcement learning, two dominant techniques ex- ist: optimization-based (Finn et al., 2017; 
Revised: Within meta-reinforcement learning, two dominant techniques exist: optimization-based (Finn et al., 2017; Rothfuss et al., 2019; Zintgraf et al., 2019;

Original: Rothfuss et al., 2019; Zintgraf et al., 2019; Stadie et al., 2018) and contextbased, which includes both recurrent architectures (Duan et al., 2016; Wang et al., 2016;
Revised: Stadie et al., 2018) and contextbased, which includes both recurrent architectures (Duan et al., 2016; Wang et al., 2016;

Original: Mishra et al., 2018) and architectures based on latent variable inference (Rakelly et al., 2019; Lee et al., 2019a; Zintgraf et al., 2020).
Revised: Mishra et al., 2018) and architectures based on latent variable inference (Rakelly et al., 2019; Lee et al., 2019a; Zintgraf et al., 2020).

Original: LILAC ﬁts into this last category within this taxonomy, but extends previous methods by considering inter-episode latent variable dynamics.
Revised: LILAC ﬁts into this last category within this taxonomy, but extends pre- vious methods by considering inter-episode latent variable dynamics.

Original: Previous embedding-based meta-RL algorithms— while able to perform online inference of latent variables and incorporate this posterior belief into action selection— do not consider how these latent variables evolve over the lifetime of the agent, as in the DP-MDP setting.
Revised: Previous embedding-based meta-RL algorithms— while able to perform online inference of latent variables and incorporate this posterior belief into action selection— do not consider how these latent variables evolve over the lifetime of the agent, as in the DP-MDP setting.

Original: The inner latent variable inference component of LILAC possesses strong similarities to the continual and lifelong learning setting (Gama et al., 2014).
Revised: The inner latent variable inference component of LILAC possesses strong similarities to the continual and lifelong learning setting (Gama et al., 2014).

Original: Indeed, whereas context-based meta-RL approaches do not consider the correlation of latent factors between episodes (beyond assuming iid draws from a shared prior) and thus are not able to share knowledge between subsequent episodes, LILAC may be interpreted as an approach towards leaning-to-lifelonglearn.
Revised: Indeed, whereas context-based meta-RL approaches do not consider the correlation of latent factors between episodes (beyond assuming iid draws from a shared prior) and thus are not able to share knowledge between subsequent episodes, LILAC may be interpreted as an approach towards leaning-to-lifelonglearn.

Original: Many continual and lifelong learning aim to learn a variety of tasks without forgetting previous tasks (Kirkpatrick et al., 2017; Zenke et al., 2017; Lopez-Paz et al., 2017;
Revised: Many continual and lifelong learning aim to learn a variety of tasks without forgetting previous tasks (Kirkpatrick et al., 2017; Zenke et al., 2017; Lopez-Paz et al., 2017;

Original: Aljundi et al., 2019; Parisi et al., 2019; Rusu et al., 2016; Shmelkov et al., 2017; Rebufﬁ et al., 2017; Shin et al., 2017).
Revised: Aljundi et al., 2019; Parisi et al., 2019; Rusu et al., 2016; Shmelkov et al., 2017; Rebufﬁ et al., 2017; Shin et al., 2017).

Original: We consider a setting where it is practical to store past experiences in a replay buffer (Rolnick et al., 2019; Finn et al., 2019).
Revised: We consider a setting where it is practical to store past experiences in a replay buffer (Rolnick et al., 2019; Finn et al., 2019).

Original: Unlike these prior works, LILAC aims to learn the dynamics associated with latent factors, and perform online inference.
Revised: Unlike these prior works, LILAC aims to learn the dynamics associated with latent factors, and perform online inference.

Original: 6. Experimental Evaluation
Revised: 6. Experimental Evaluation

Original: In our experiments, we aim to address our central hypothesis: by leveraging our latent variable model, our approach can make learning under persistent non-stationarity both effective and efﬁcient.
Revised: In our experiments, we aim to address our central hypothesis: by leveraging our latent variable model, our approach can make learning under persistent non-stationarity both effective and efﬁcient.

Original: Environments.
Revised: Environments.

Original: We construct four continuous control environments with persistent varying sources of change in the reward and/or dynamics.
Revised: We construct four continuous control environments with persistent varying sources of change in the reward and/or dynamics.

Original: These environments are designed such that the policy needs to change in order to achieve good performance.
Revised: These environments are designed such that the policy needs to change in order to achieve good performance.

Original: The ﬁrst is derived from the simulated
Revised: The ﬁrst is derived from the simulated

Original: Sawyer reaching task in the Meta-World benchmark (Yu et al., 2019), in which the target position is not observed and moves between episodes.
Revised: Sawyer reaching task in the Meta-World benchmark (Yu et al., 2019), in which the target position is not observed and moves between episodes.

Original: In the second environment based on Half-Cheetah from OpenAI Gym (Brockman et al., 2016), we consider changes in the direction and magnitude of wind forces on the agent, and changes in the target velocity.
Revised: In the second environment based on Half-Cheetah from OpenAI Gym (Brockman et al., 2016), we consider changes in the direction and magnitude of wind forces on the agent, and changes in the target velocity.

Original: We next consider the 8-DoF minitaur environment (Tan et al., 2018) and vary the mass of the agent between episodes, representative of a varying payload.
Revised: We next consider the 8-DoF minitaur environment (Tan et al., 2018) and vary the mass of the agent between episodes, representative of a varying payload.

Original: Finally, we construct a 2D navigation task in an inﬁnite, non-episodic environment with non-stationary dynamics which we call 2D Open World.
Revised: Finally, we construct a 2D navigation task in an inﬁnite, non-episodic environment with non-stationary dynamics which we call 2D Open World.

Original: The agent’s goal is to collect food pellets and to avoid other objects and obstacles, whilst subjected to unknown perturbations that vary on an episodic schedule.
Revised: The agent’s goal is to collect food pellets and to avoid other objects and obstacles, whilst subjected to unknown perturbations that vary on an episodic schedule.

Original: These environments are illustrated in Figure 4.
Revised: These environments are illustrated in Figure 4.

Original: For full environment details, see
Revised: For full environment details, see

Original: Appendix A.
Revised: Appendix A.

Original: Comparisons.
Revised: Comparisons.

Original: We compare our approach to standard softactor critic (SAC) (Haarnoja et al., 2018), which corresponds to our method without any latent variables, allowing us to evaluate the performance of off-policy algorithms amid non-stationarity.
Revised: We compare our approach to standard softactor critic (SAC) (Haarnoja et al., 2018), which corresponds to our method without any latent variables, allowing us to evaluate the performance of off-policy algorithms amid non-stationarity.

Original: We also compare to stochastic latent actor-critic (SLAC)
Revised: We also compare to stochastic latent actor-critic (SLAC)

Original: (Lee et al., 2019a), which learns to model partially observed environments with a latent variable model but does not address inter-episode non-stationarity.
Revised: (Lee et al., 2019a), which learns to model partially observed environments with a latent variable model but does not address inter-episode non-stationarity.

Original: This comparison allows us to evaluate the importance of modeling non-stationarity between episodes.
Revised: This comparison allows us to evaluate the importance of modeling non-stationarity between episodes.

Original: Finally, we include proximal policy optimization (PPO) (Schulman et al., 2017) as a comparison to on-policy RL.
Revised: Finally, we include proximal policy optimization (PPO) (Schulman et al., 2017) as a comparison to on-policy RL.

Original: Since the tasks in the Sawyer and Half-Cheetah domains involve goal reaching, we can obtain an oracle by training a goal-conditioned
Revised: Since the tasks in the Sawyer and Half-Cheetah domains involve goal reaching, we can obtain an oracle by training a goal-conditioned

Original: SAC policy, i.e. with the true goal concatenated to the observation.
Revised: SAC policy, i.e. with the true goal concatenated to the observation.

Original: We provide this comparison to help contextualize the performance of our method against other algorithms.
Revised: We provide this comparison to help contextualize the performance of our method against other algorithms.

Original: For all hyperparameter details, see Appendix B.
Revised: For all hyperparameter details, see Appendix B.

Original: Results.
Revised: Results.

Original: Our experimental results are shown in Figure 5.
Revised: Our experimental results are shown in Figure 5.

Original: Since on-policy algorithms tend to have worse sample complexity, we run PPO for 10 million environment steps and plot only the asymptotic returns.
Revised: Since on-policy algorithms tend to have worse sample complexity, we run PPO for 10 million environment steps and plot only the asymptotic returns.

Original: In all domains, LILAC attains higher and more stable returns compared to SAC,
Revised: In all domains, LILAC attains higher and more stable returns compared to SAC,

Original: SLAC, and PPO.
Revised: SLAC, and PPO.

Original: Since SAC amortizes experience collected across episodes into a single replay buffer, we observe that the algorithm converges to an averaged behavior.
Revised: Since SAC amortizes experience collected across episodes into a single replay buffer, we observe that the algorithm converges to an averaged behavior.

Original: Meanwhile, SLAC does not have the mechanism to model nonstationarity across episodes, and has to infer the unknown dynamics and reward from the initial steps taken during each episode, which the algorithm is not very successful at.
Revised: Meanwhile, SLAC does not have the mechanism to model nonstationarity across episodes, and has to infer the unknown dynamics and reward from the initial steps taken during each episode, which the algorithm is not very successful at.

Original: Due to the cyclical nature of the tasks, the learned behavior of
Revised: Due to the cyclical nature of the tasks, the learned behavior of

Original: SLAC results in oscillating returns across tasks.
Revised: SLAC results in oscillating returns across tasks.

Original: Similarly,
Revised: Similarly,

Original: PPO cannot adapt to per-episode changes in the environment and ultimately converges to learning an average policy.
Revised: PPO cannot adapt to per-episode changes in the environment and ultimately converges to learning an average policy.

Original: In contrast to these methods, LILAC infers how the environment changes in future episodes and steadily maintains high rewards over the training procedure, despite experiencing persistent shifts in the environment in each episode.
Revised: In contrast to these methods, LILAC infers how the environment changes in future episodes and steadily maintains high rewards over the training procedure, despite experiencing persistent shifts in the environment in each episode.

Original: Further, LILAC can learn under simultaneous shifts in both dynamics and rewards, veriﬁed by the HC WindVel results.
Revised: Further, LILAC can learn under simultaneous shifts in both dynamics and rewards, veriﬁed by the HC WindVel results.

Original: Notably, LILAC also adeptly handles shifts in the 2D Open
Revised: Notably, LILAC also adeptly handles shifts in the 2D Open

Original: World environment without episodic resets.
Revised: World environment without episodic resets.

Original: A partial snapshot of the agent’s lifetime from this task is visualized in
Revised: A partial snapshot of the agent’s lifetime from this task is visualized in

Original: Figure 4.
Revised: Figure 4.

Original: Rate of environment shift.
Revised: Rate of environment shift.

Original: We next evaluate whether
Revised: We next evaluate whether

Original: LILAC can handle varying rates of non-stationarity.
Revised: LILAC can handle varying rates of non-stationarity.

Original: To do so, we use the Sawyer reaching domain, where the goal moves along a ﬁxed-radius circle, and vary the step size along the circle to generate environments that shift at different speeds.
Revised: To do so, we use the Sawyer reaching domain, where the goal moves along a ﬁxed-radius circle, and vary the step size along the circle to generate environments that shift at different speeds.

Original: As depicted in Figure 6, LILAC’s performance is largely independent of the environment’s rate of change.
Revised: As depicted in Figure 6, LILAC’s performance is largely independent of the environment’s rate of change.

Original: We also evaluate LILAC under stationary conditions, i.e. with a ﬁxed goal, and LILAC achieves the same performance as
Revised: We also evaluate LILAC under stationary conditions, i e. with a ﬁxed goal and LILAC achieves the same performance as

Original: SAC, thus retaining the ability to learn as effectively as SAC in a ﬁxed environment.
Revised: SAC thus retaining the ability to learn as effectively as SAC in a ﬁxed environment.

Original: These results demonstrate LILAC’s efﬁcacy under a range of rates of non-stationarity, including the stationary case.
Revised: These results demonstrate LILAC’s efﬁcacy under a range of rates of non-stationarity, including the stationary case.

Original: The gap in LILAC’s performance between the non-stationary and stationary cases is likely due to imprecise estimates of future environment conditions given by the prior p φ ( z (cid:48) | z ) .
Revised: The gap in LILAC’s performance between the non-stationary and stationary cases is likely due to imprecise estimates of future environment conditions given by the prior p φ ( z (cid:48) | z ) .

Original: Currently, the executed policy uses ﬁxed z given by the prior for the entire episode, but a natural extension that may improve performance is updating z during each episode by encoding partial trajectories with the inference network.
Revised: Currently, the executed policy uses ﬁxed z given by the prior for the entire episode but a natural extension that may improve performance is updating z during each episode by encoding partial trajectories with the inference network.

Original: 7. Conclusion
Revised: 7. Conclusion

Original: We considered the problem of reinforcement learning with lifelong non-stationarity, a problem which we believe is critical to reinforcement learning systems operating in the real world.
Revised: We considered the problem of reinforcement learning with lifelong non-stationarity, a problem which we believe is critical to reinforcement learning systems operating in the real world.

Original: This problem is at the intersection of reinforcement learning under partial observability (i.e. POMDPs) and online learning; hence we formalized the problem as a special case of a POMDP that is also signiﬁcantly more tractable.
Revised: This problem is at the intersection of reinforcement learning under partial observability (i e. POMDPs) and online learning; hence we formalized the problem as a special case of a POMDP that is also signiﬁcantly more tractable.

Original: We derive a graphical model underlying this problem setting, and utilize it to derive our approach under the formalism of reinforcement learning as probabilistic inference (Levine, 2018).
Revised: We derive a graphical model underlying this problem setting, and utilize it to derive our approach under the formalism of reinforcement learning as probabilistic inference (Levine, 2018).

Original: Our method leverages this latent variable model to model the change in the environment, and conditions the policy and critic on the inferred values of these latent variables.
Revised: Our method leverages this latent variable model to model the change in the environment, and conditions the policy and critic on the inferred values of these latent variables.

Original: On a variety of challenging continuous control tasks with signiﬁcant non-stationarity, we observe that our approach leads to substantial improvement compared to state-of-the-art reinforcement learning methods.
Revised: On a variety of challenging continuous control tasks with signiﬁcant non-stationarity, we observe that our approach leads to substantial improvement compared to state-of-the-art reinforcement learning methods.

Original: While the DP-MDP formulation represents a strict generalization of the commonly-considered meta-reinforcement learning settings (typically, a BAMDP (Zintgraf et al., , it is still somewhat limited in its generality.
Revised: While the DP-MDP formulation represents a strict generalization of the commonly-considered meta-reinforcement learning settings (typically, a BAMDP (Zintgraf et al., 2020)), it is still somewhat limited in its generality.

Original: In particular, the assumption of task parameters shifting between episodes, but never during them, presents a possibly unrealistic limitation.
Revised: In particular, the assumption of task parameters shifting between episodes, but never during them, presents a possibly unrealistic limitation.

Original: While relaxing this assumption leads, in the worst case, to a POMDP, there is potentially additional structure that may be exploited under the HM-MDP (Choi et al., 2000) assumption of infrequent, discrete, unobserved shifts in the task parameters.
Revised: While relaxing this assumption leads, in the worst case, to a POMDP, there is potentially additional structure that may be exploited under the HM-MDP (Choi et al., 2000) assumption of infrequent, discrete, unobserved shifts in the task parameters.

Original: In particular, this notion of infrequent, discrete shifts underlies the changepoint detection literature (Adams & MacKay, 2007; Fearnhead & Liu, 2007).
Revised: In particular, this notion of infrequent, discrete shifts underlies the changepoint detection literature (Adams & MacKay, 2007; Fearnhead & Liu, 2007).

Original: Previous work both within sequential decision making in changing environments (Da Silva et al., 2006; Hadoux et al., 2014; Banerjee et al., 2017) and meta-learning within changing data streams (Harrison et al., 2019) may enable a version of LILAC capable of handling unobserved changepoints, and this setting is likely a fruitful direction for future research.
Revised: Previous work both within sequential decision making in changing environments (Da Silva et al., 2006; Hadoux et al., 2014; Banerjee et al., 2017) and meta-learning within changing data streams (Harrison et al., 2019) may enable a version of LILAC capable of handling unobserved changepoints, and this setting is likely a fruitful direction for future research.

Original: A. Environment Details
Revised: A. Environment Details

Original: Below, we provide environment details for each of the four experimental domains.
Revised: Below, we provide environment details for each of the four experimental domains.

Original: A.1. Sawyer Reaching
Revised: A.1. Sawyer Reaching

Original: In this environment, which is based on the simulated Sawyer reaching task in the Meta-World suite (Yu et al., 2019), the goal is to reach a particular position.
Revised: In this environment, which is based on the simulated Sawyer reaching task in the Meta-World suite (Yu et al., 2019), the goal is to reach a particular position.

Original: The target position, which is unobserved throughout, moves after each episode.
Revised: The target position, which is unobserved throughout, moves after each episode.

Original: The episodes are 150 timesteps long, and the state is the position of the end-effector in ( x, y, z ) coordinate space.
Revised: The episodes are 150 timesteps long, and the state is the position of the end-effector in ( x, y, z ) coordinate space.

Original: The actions correspond to changes in end-effector positions.
Revised: The actions correspond to changes in end-effector positions.

Original: The reward is deﬁned as
Revised: The reward is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where s g at episode i is deﬁned as
Revised: where s g at episode i is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: In other words, the sequence of goals is deﬁned by a circle in the xy -plane.
Revised: In other words, the sequence of goals is deﬁned by a circle in the xy -plane.

Original: For the oracle comparison, the sequence of goals and the reward function are the same, except the state observation here is the concatenation of the end-effector position and the goal position s g .
Revised: For the oracle comparison, the sequence of goals and the reward function are the same, except the state observation here is the concatenation of the end-effector position and the goal position s g .

Original: A.2. Half-Cheetah Vel
Revised: A.2. Half-Cheetah Vel

Original: This environment builds off of the Half-Cheetah environment from OpenAI Gym (Brockman et al., 2016).
Revised: This environment builds off of the Half-Cheetah environment from OpenAI Gym (Brockman et al., 2016).

Original: In this domain, the agent must reach a target velocity in the x -direction, which varies across episodes, i.e., the reward is
Revised: In this domain, the agent must reach a target velocity in the x -direction, which varies across episodes, i.e., the reward is

Original: [Equation]
Revised: [Equation]

Original: where v s is the observed velocity of the agent.
Revised: where v s is the observed velocity of the agent.

Original: The state consists of the position and velocity of the agent’s center of mass and the angular position and angular velocity of each of its six joints, and actions correspond to torques applied to each of the six joints.
Revised: The state consists of the position and velocity of the agent’s center of mass and the angular position and angular velocity of each of its six joints, and actions correspond to torques applied to each of the six joints.

Original: The target velocity v g varies according to a sine function, i.e., the target velocity for episode i is 
Revised: The target velocity v g varies according to a sine function, i.e., the target velocity for episode i is v

Original: [Equation]
Revised: [Equation]

Revised: 1 . 5 + 1 .

Revised: sin(0 .

Revised: 5 · i ) .

Original: For the oracle comparison, the target velocity v g is appended to the state observation.
Revised: For the oracle comparison, the target velocity v g is appended to the state observation.

Original: Each episode, across all comparisons, is 50 timesteps long.
Revised: Each episode, across all comparisons, is 50 timesteps long.

Original: A.3. Half-Cheetah Wind+Vel
Revised: A.3. Half-Cheetah Wind+Vel

Original: In this variant of Half-Cheetah Vel, the agent is additionally subjected to varying wind forces.
Revised: In this variant of Half-Cheetah Vel, the agent is additionally subjected to varying wind forces.

Original: The force for each episode is deﬁned by
Revised: The force for each episode is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: and is applied constantly along the x -direction throughout the episode.
Revised: and is applied constantly along the x -direction throughout the episode.

Original: A.4. Minitaur Mass
Revised: A.4. Minitaur Mass

Original: For this last domain, we use the simulated Minitaur environment developed by Tan et al.
Revised: For this last domain, we use the simulated Minitaur environment developed by Tan et al.

Original: We induce non-stationarity by varying the mass of the agent between episodes akin to increasing and decreasing payloads.
Revised: We induce non-stationarity by varying the mass of the agent between episodes akin to increasing and decreasing payloads.

Original: Speciﬁcally, the mass at each episode is
Revised: Speciﬁcally, the mass at each episode is

Original: [Equation]
Revised: [Equation]

Original: The reward is deﬁned by
Revised: The reward is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst two terms correspond to the velocity reward, which encourages the agent to run close to a target velocity of 0 .
Revised: where the ﬁrst two terms correspond to the velocity reward, which encourages the agent to run close to a target velocity of 0 .

Original: 5 m/s, and the last term corresponds to an acceleration penalty deﬁned by the last three actions taken by the agent.
Revised: 5 m/s, and the last term corresponds to an acceleration penalty deﬁned by the last three actions taken by the agent.

Original: The state includes the angles, velocities, and torques of all eight motors, and the action is the target motor angle for each motor.
Revised: The state includes the angles, velocities, and torques of all eight motors, and the action is the target motor angle for each motor.

Original: Each episode is 100 timesteps long.
Revised: Each episode is 100 timesteps long.

Original: B. Experimental and Hyperparameter Details
Revised: B. Experimental and Hyperparameter Details

Original: In this section, we provide details of the hyperparameters used for each method.
Revised: In this section, we provide details of the hyperparameters used for each method.

Original: B.1. LILAC (Ours)
Revised: B.1. LILAC (Ours)

Original: Latent space.
Revised: Latent space.

Original: For our method, we use a latent space size of 8 in Sawyer Reaching, and size of 40 in the other experiments:
Revised: For our method, we use a latent space size of 8 in Sawyer Reaching, and size of 40 in the other experiments:

Original: Half-Cheetah Vel, Half-Cheetah Wind+Vel, and Minitaur Mass.
Revised: Half-Cheetah Vel, Half-Cheetah Wind+Vel, and Minitaur Mass.

Original: Inference and decoder networks.
Revised: Inference and decoder networks.

Original: The inference and decoder networks are MLPs with 2 fully-connected layers of size 64 in Sawyer Reaching; 1 fully-connected layer of size 512 in Half-Cheetah Vel and Half-Cheetah Wind+Vel; andfully-connected layers of size 512 in Minitaur Mass.
Revised: The inference and decoder networks are MLPs with 2 fully-connected layers of size 64 in Sawyer Reaching; 1 fully-connected layer of size 512 in Half-Cheetah Vel and Half-Cheetah Wind+Vel; andfully-connected layers of size 512 in Minitaur Mass.

Original: Policy and critic networks.
Revised: Policy and critic networks.

Original: The policy and critic networks are MLPs with 3 fully-connected layers of size 256 in the Sawyer
Revised: The policy and critic networks are MLPs with 3 fully-connected layers of size 256 in the Sawyer

Original: Reaching experiment; and 2 fully-connected layers of size 256 in the other experiments.
Revised: Reaching experiment; and 2 fully-connected layers of size 256 in the other experiments.

Original: Training.
Revised: Training.

Original: For each training iteration, we sample a batch of 32 episodes and from each episode, we sample 8 tuples of transitions and rewards ( s , a , s (cid:48) , r ) .
Revised: For each training iteration, we sample a batch of 32 episodes and from each episode, we sample 8 tuples of transitions and rewards ( s , a , s (cid:48) , r ) .

Original: The training objective of the inference network is a linear combination of the reconstruction loss, critic loss, and KL divergence from the learned prior:
Revised: The training objective of the inference network is a linear combination of the reconstruction loss, critic loss, and KL divergence from the learned prior:

Original: [Equation]
Revised: [Equation]

Original: For the Sawyer Reaching experiment, β 1 and β 2 are 
Revised: For the Sawyer Reaching experiment, β 1 and β 2 are β 1 = (cid:40) 0 , iter <1 , iter ≥β 2 = (cid:40) 0 , iter <min(1 e-

Original: [Equation] iter[Equation] ≥
Revised: 6 · ( iter − 10000) , 1) , iter ≥

Original: For Half-Cheetah Vel and Half-Cheetah Wind+Vel, β 1 and β 2 are β 1 = (cid:40) 0 , iter <1 , iter ≥β 2 = (cid:40) 0 , iter <min(1 e-
Revised: For Half-Cheetah Vel and Half-Cheetah Wind+Vel, β 1 and β 2 are β 1 = (cid:40) 0 , iter <1 , iter ≥β 2 = (cid:40) 0 , iter <min(1 e-

Original: 6 · ( iter − 10000) , 1) , iter ≥
Revised: 6 · ( iter − 10000) , 1) , iter ≥

Original: Finally, for the Minitaur Mass experiment, β 1 and β 2 are β 1 = (cid:40) 0 , iter <1 , iter ≥β 2 = (cid:40) 0 , iter <1 e- 6 , iter ≥
Revised: Finally, for the Minitaur Mass experiment, β 1 and β 2 are β 1 = (cid:40) 0 , iter <1 , iter ≥β 2 = (cid:40) 0 , iter <1 e- 6 , iter ≥

Original: Critic (Lee et al., 2019a)
Revised: B.2. Stochastic Latent Actor-Critic (Lee et al., 2019a)

Original: Latent space.
Revised: Latent space.

Original: SLAC factorizes its per-timestep latent variable z t into two stochastic layers z 1 t and z 2 t , i.e. p ( z t )
Revised: SLAC factorizes its per-timestep latent variable z t into two stochastic layers z 1 t and z 2 t , i.e. p ( z t )

Original: =
Revised: =

Original: p ( z 2 t | z 1 t ) p ( z 1 t ) .
Revised: p ( z 2 t | z 1 t ) p ( z 1 t ) .

Original: In the Sawyer Reaching experiment, the size of z 1 t is 16 and the size of z 2 t is 8 .
Revised: In the Sawyer Reaching experiment, the size of z 1 t is 16 and the size of z 2 t is 8 .

Original: In all other experiments, the size of z 1 t is 64 and the size of z 2 t is 32 .
Revised: In all other experiments, the size of z 1 t is 64 and the size of z 2 t is 32 .

Original: Inference and decoder networks.
Revised: Inference and decoder networks.

Original: The inference and decoder networks are MLPs with 2 fully-connected layers of size 64 in Sawyer Reaching; 1 fully-connected layer of size 512 in Half-Cheetah Vel and Half-Cheetah Wind+Vel; andfully-connected layers of size 512 in Minitaur Mass.
Revised: The inference and decoder networks are MLPs with 2 fully-connected layers of size 64 in Sawyer Reaching; 1 fully-connected layer of size 512 in Half-Cheetah Vel and Half-Cheetah Wind+Vel; andfully-connected layers of size 512 in Minitaur Mass.

Original: Policy and critic networks.
Revised: Policy and critic networks.

Original: The policy and critic networks are MLPs with 3 fully-connected layers of size 256 in the Sawyer
Revised: The policy and critic networks are MLPs with 3 fully-connected layers of size 256 in the Sawyer

Original: Reaching experiment; and 2 fully-connected layers of size 256 in the other experiments.
Revised: Reaching experiment; and 2 fully-connected layers of size 256 in the other experiments.

Original: B.3. Soft Actor-Critic (
Revised: B.3. Soft Actor-Critic (Haarnoja

Original: Haarnoja et al.,2018)
Revised: et al.,

Original: Policy and critic networks.
Revised: Policy and critic networks.

Original: The policy and critic networks are MLPs with 3 fully-connected layers of size 256 in the Sawyer
Revised: The policy and critic networks are MLPs with 3 fully-connected layers of size 256 in the Sawyer

Original: Reaching experiment; and 2 fully-connected layers of size 256 in the other experiments.
Revised: Reaching experiment; and 2 fully-connected layers of size 256 in the other experiments.