Original: L EARNING TO A DAPT IN D YNAMIC , R EAL -W ORLD E NVIRONMENTS THROUGH M ETA -R EINFORCEMENT L EARNING
Revised: L EARNING TO A DAPT IN D YNAMIC , R EAL -W ORLD E NVIRONMENTS THROUGH M ETA -R EINFORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Although reinforcement learning methods can achieve impressive results in simulation, the real world presents two major challenges: generating samples is exceedingly expensive, and unexpected perturbations or unseen situations cause proﬁcient but specialized policies to fail at test time.
Revised: Although reinforcement learning methods can achieve impressive results in simulation, the real world presents two major challenges: generating samples is exceedingly expensive, and unexpected perturbations or unseen situations cause proﬁcient but specialized policies to fail at test time.

Original: Given that it is impractical to train separate policies to accommodate all situations the agent may see in the real world, this work proposes to learn how to quickly and effectively adapt online to new tasks.
Revised: Given that it is impractical to train separate policies to accommodate all situations the agent may see in the real world, this work proposes to learn how to quickly and effectively adapt online to new tasks.

Original: To enable sample-efﬁcient learning, we consider learning online adaptation in the context of model-based reinforcement learning.
Revised: To enable sample-efﬁcient learning, we consider learning online adaptation in the context of model-based reinforcement learning.

Original: Our approach uses meta-learning to train a dynamics model prior such that, when combined with recent data, this prior can be rapidly adapted to the local context.
Revised: Our approach uses meta-learning to train a dynamics model prior such that, when combined with recent data, this prior can be rapidly adapted to the local context.

Original: Our experiments demonstrate online adaptation for continuous control tasks on both simulated and real-world agents.
Revised: Our experiments demonstrate online adaptation for continuous control tasks on both simulated and real-world agents.

Original: We ﬁrst show simulated agents adapting their behavior online to novel terrains, crippled body parts, and highly-dynamic environments.
Revised: We ﬁrst show simulated agents adapting their behavior online to novel terrains, crippled body parts, and highly-dynamic environments.

Original: We also illustrate the importance of incorporating online adaptation into autonomous agents that operate in the real world by applying our method to a real dynamic legged millirobot.
Revised: We also illustrate the importance of incorporating online adaptation into autonomous agents that operate in the real world by applying our method to a real dynamic legged millirobot.

Original: We demonstrate the agent’s learned ability to quickly adapt online to a missing leg, adjust to novel terrains and slopes, account for miscalibration or errors in pose estimation, and compensate for pulling payloads.
Revised: We demonstrate the agent’s learned ability to quickly adapt online to a missing leg, adjust to novel terrains and slopes, account for miscalibration or errors in pose estimation, and compensate for pulling payloads.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Both model-based and model-free reinforcement learning (RL) methods generally operate in one of two regimes: all training is performed in advance, producing a model or policy that can be used at test-time to make decisions in settings that approximately match those seen during training; or, training is performed online (e.g., as in the case of online temporal-difference learning), in which case the agent can slowly modify its behavior as it interacts with the environment.
Revised: Both model-based and model-free reinforcement learning (RL) methods generally operate in one of two regimes: all training is performed in advance, producing a model or policy that can be used at test-time to make decisions in settings that approximately match those seen during training; or, training is performed online (e.g., as in the case of online temporal-difference learning), in which case the agent can slowly modify its behavior as it interacts with the environment.

Original: However, in both of these cases, dynamic changes such as failure of a robot’s components, encountering a new terrain, environmental factors such as lighting and wind, or other unexpected perturbations, can cause the agent to fail.
Revised: However, in both of these cases, dynamic changes such as failure of a robot’s components, encountering a new terrain, environmental factors such as lighting and wind, or other unexpected perturbations, can cause the agent to fail.

Original: In contrast, humans can rapidly adapt their behavior to unseen physical perturbations and changes in their dynamics (Braun et al., 2009): adults can learn to walk on crutches in just a few seconds, people can adapt almost instantaneously to picking up an object that is unexpectedly heavy, and children that can walk on carpet and grass can quickly ﬁgure out how to walk on ice without having to relearn how to walk.
Revised: In contrast, humans can rapidly adapt their behavior to unseen physical perturbations and changes in their dynamics (Braun et al., 2009): adults can learn to walk on crutches in just a few seconds, people can adapt almost instantaneously to picking up an object that is unexpectedly heavy, and children that can walk on carpet and grass can quickly ﬁgure out how to walk on ice without having to relearn how to walk.

Original: How is this possible?
Revised: How is this possible?

Original: If an agent has encountered a large number of perturbations in the past, it can in principle use that experience to learn how to adapt .
Revised: If an agent has encountered a large number of perturbations in the past, it can in principle use that experience to learn how to adapt .

Original: In this work, we propose a meta-learning approach for learning online adaptation.
Revised: In this work, we propose a meta-learning approach for learning online adaptation.

Original: Motivated by the ability to tackle real-world applications, we speciﬁcally develop a model-based meta-reinforcement learning algorithm.
Revised: Motivated by the ability to tackle real-world applications, we speciﬁcally develop a model-based meta-reinforcement learning algorithm.

Original: In this setting, data for updating the model is readily available at every timestep in the form of recent experiences.
Revised: In this setting, data for updating the model is readily available at every timestep in the form of recent experiences.

Original: But more crucially, the meta-training process for training such an adaptive model can be much more sample efﬁcient than model-free meta-RL approaches (Duan et al., 2016;
Revised: But more crucially, the meta-training process for training such an adaptive model can be much more sample efﬁcient than model-free meta-RL approaches (Duan et al., 2016;

Original: Wang et al., 2016; Finn et al., 2017).
Revised: Wang et al., 2016; Finn et al., 2017).

Original: Further, our approach foregoes the episodic framework on which model-free meta-RL approaches rely on, where tasks are pre-deﬁned to be different rewards or environments, and tasks exist at the trajectory level only.
Revised: Further, our approach foregoes the episodic framework on which model-free meta-RL approaches rely on, where tasks are pre-deﬁned to be different rewards or environments, and tasks exist at the trajectory level only.

Original: Instead, our method considers each timestep to potentially be a new “task," where any detail or setting could have changed at any timestep.
Revised: Instead, our method considers each timestep to potentially be a new “task, " where any detail or setting could have changed at any timestep.

Original: This view induces a more general meta-RL problem setting by allowing the notion of a task to represent anything from existing in a different part of the state space, to experiencing disturbances, or attempting to achieve a new goal.
Revised: This view induces a more general meta-RL problem setting by allowing the notion of a task to represent anything from existing in a different part of the state space, to experiencing disturbances, or attempting to achieve a new goal.

Original: Learning to adapt a model alleviates a central challenge of model-based reinforcement learning: the problem of acquiring a global model that is accurate throughout the entire state space.
Revised: Learning to adapt a model alleviates a central challenge of model-based reinforcement learning: the problem of acquiring a global model that is accurate throughout the entire state space.

Original: Furthermore, even if it were practical to train a globally accurate dynamics model, the dynamics inherently change as a function of uncontrollable and often unobservable environmental factors, such as those mentioned above.
Revised: Furthermore, even if it were practical to train a globally accurate dynamics model, the dynamics inherently change as a function of uncontrollable and often unobservable environmental factors, such as those mentioned above.

Original: If we have a model that can adapt online, it need not be perfect everywhere a priori.
Revised: If we have a model that can adapt online, it need not be perfect everywhere a priori.

Original: This property has previously been exploited by adaptive control methods (Åström and Wittenmark, 2013;
Revised: This property has previously been exploited by adaptive control methods (Åström and Wittenmark, 2013;

Original: Sastry and Isidori, 1989); but, scaling such methods to complex tasks and nonlinear systems is exceptionally difﬁcult.
Revised: Sastry and Isidori, 1989); but, scaling such methods to complex tasks and nonlinear systems is exceptionally difﬁcult.

Original: Even when working with deep neural networks, which have been used to model complex nonlinear systems (Kurutach et al., 2018), it is exceptionally difﬁcult to enable adaptation, since such models typically require large amounts of data and many gradient steps to learn effectively.
Revised: Even when working with deep neural networks, which have been used to model complex nonlinear systems (Kurutach et al., 2018), it is exceptionally difﬁcult to enable adaptation, since such models typically require large amounts of data and many gradient steps to learn effectively.

Original: By speciﬁcally training a neural network model to require only a small amount of experience to adapt, we can enable effective online adaptation in complex environments while putting less pressure on needing a perfect global model.
Revised: By speciﬁcally training a neural network model to require only a small amount of experience to adapt, we can enable effective online adaptation in complex environments while putting less pressure on needing a perfect global model.

Original: The primary contribution of our work is an efﬁcient meta reinforcement learning approach that achieves online adaptation in dynamic environments.
Revised: The primary contribution of our work is an efﬁcient meta reinforcement learning approach that achieves online adaptation in dynamic environments.

Original: To the best knowledge of the authors, this is the ﬁrst meta-reinforcement learning algorithm to be applied in a real robotic system.
Revised: To the best knowledge of the authors, this is the ﬁrst meta-reinforcement learning algorithm to be applied in a real robotic system.

Original: Our algorithm efﬁciently trains a global model that is capable to use its recent experiences to quickly adapt, achieving fast online adaptation in dynamic environments.
Revised: Our algorithm efﬁciently trains a global model that is capable to use its recent experiences to quickly adapt, achieving fast online adaptation in dynamic environments.

Original: We evaluate two versions of our approach, recurrence-based adaptive learner (ReBAL) and gradient-based adaptive learner (GrBAL) on stochastic and simulated continuous control tasks with complex contact dynamics (Fig. 2).
Revised: We evaluate two versions of our approach, recurrence-based adaptive learner (ReBAL) and gradient-based adaptive learner (GrBAL) on stochastic and simulated continuous control tasks with complex contact dynamics (Fig. 2).

Original: In our experiments, we show(1) a quadrupedal “ant” adapting to the failure of different legs, as well as a “half-cheetah" robot adapting to the failure off different joints, navigating terrains with different slopes, and walking on ﬂoating platforms of varying buoyancy.
Revised: In our experiments, we show a quadrupedal “ant” adapting to the failure of different legs, as well as a “half-cheetah" robot adapting to the failure off different joints, navigating terrains with different slopes, and walking on ﬂoating platforms of varying buoyancy.

Original: Our model-based meta RL method attains substantial improvement over prior approaches, including standard model-based methods, online model-adaptive methods, model-free methods, and prior meta-reinforcement learning methods, when trained with similar amounts of data.
Revised: Our model-based meta RL method attains substantial improvement over prior approaches, including standard model-based methods, online model-adaptive methods, model-free methods, and prior meta-reinforcement learning methods, when trained with similar amounts of data.

Original: In all experiments, meta-training across multiple tasks is sample efﬁcient, using only the equivalent of 1 . 5 − 3 hours of real-world experience, roughly 10 × less than what model-free methods require to learn a single task.
Revised: In all experiments, meta-training across multiple tasks is sample efﬁcient, using only the equivalent of 1 . 5 − 3 hours of real-world experience, roughly 10 × less than what model-free methods require to learn a single task.

Original: Finally, we demonstrate GrBAL on a real dynamic legged millirobot (see Fig 2).
Revised: Finally, we demonstrate GrBAL on a real dynamic legged millirobot (see Fig 2).

Original: To highlight not only the sample efﬁciency of our meta model-based reinforcement learning approach, but also the importance of fast online adaptation in the real world, we show the agent’s learned ability to adapt online to tasks such as a missing leg, novel terrains and slopes, miscalibration or errors in pose estimation, and new payloads to be pulled.
Revised: To highlight not only the sample efﬁciency of our meta model-based reinforcement learning approach, but also the importance of fast online adaptation in the real world, we show the agent’s learned ability to adapt online to tasks such as a missing leg, novel terrains and slopes, miscalibration or errors in pose estimation, and new payloads to be pulled.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Advances in learning control policies have shown success on numerous complex and high dimensional tasks (Schulman et al., 2015;
Revised: Advances in learning control policies have shown success on numerous complex and high dimensional tasks (Schulman et al., 2015;

Original: Lillicrap et al., 2015; Mnih et al., 2015; Levine et al., 2016; Silver et al., 2017).
Revised: Lillicrap et al., 2015; Mnih et al., 2015; Levine et al., 2016; Silver et al., 2017).

Original: While reinforcement learning algorithms provide a framework for learning new tasks, they primarily focus on mastery of individual skills, rather than generalizing and quickly adapting to new scenarios.
Revised: While reinforcement learning algorithms provide a framework for learning new tasks, they primarily focus on mastery of individual skills, rather than generalizing and quickly adapting to new scenarios.

Original: Furthermore, model-free approaches (Peters and Schaal, 2008) require large amounts of system interaction to learn successful control policies, which often makes them impractical for real-world systems.
Revised: Furthermore, model-free approaches (Peters and Schaal, 2008) require large amounts of system interaction to learn successful control policies, which often makes them impractical for real-world systems.

Original: In contrast, model-based methods attain superior sample efﬁciency by ﬁrst learning a model of system dynamics, and then using that model to optimize a policy (Deisenroth et al., 2013; Lenz et al., 2015; Levine et al., 2016;
Revised: In contrast, model-based methods attain superior sample efﬁciency by ﬁrst learning a model of system dynamics, and then using that model to optimize a policy (Deisenroth et al., 2013; Lenz et al., 2015; Levine et al., 2016;

Original: Nagabandi et al., 2017b).
Revised: Nagabandi et al., 2017b; Williams et al., 2017; Chua et al., 2018).

Original: A key challenge with model-based RL approaches is the difﬁculty of learning a global model that is accurate for the entire state space.
Revised: A key challenge with model-based RL approaches is the difﬁculty of learning a global model that is accurate for the entire state space.

Original: Our approach alleviates the need to learn a single global model by allowing the model to be adapted automatically to different scenarios online based on recent observations.
Revised: Our approach alleviates the need to learn a single global model by allowing the model to be adapted automatically to different scenarios online based on recent observations.

Original: Prior online adaptation approaches (Tanaskovic et al., 2013; Aswani et al., 2012) have aimed to learn an approximate global model and then adapt it at test time.
Revised: Prior online adaptation approaches (Tanaskovic et al., 2013; Aswani et al., 2012) have aimed to learn an approximate global model and then adapt it at test time.

Original: Dynamic evaluation algorithms (Rei, 2015; Krause et al., 2017; 2016; Fortunato et al., 2017), for example, learn an approximate global distribution at training time and adapt those model parameters at test time to ﬁt the current local distribution via gradient descent.
Revised: Dynamic evaluation algorithms (Rei, 2015; Krause et al., 2017; 2016; Fortunato et al., 2017), for example, learn an approximate global distribution at training time and adapt those model parameters at test time to ﬁt the current local distribution via gradient descent.

Revised: There exists extensive prior work on adapting inverse dynamics models (Kelouwani et al., 2012; Underwood and Husain, 2010), also closely related to online system identiﬁcation (Manganiello et al., 2014).

Original: Such work in model adaptation (Levine and Koltun, 2013; Gu et al., 2016; Fu et al., 2015; Weinstein and Botvinick, 2017) has shown that a perfect global model is not necessary, and prior knowledge can be ﬁne-tuned to handle small changes.
Revised: Such work in model adaptation (Levine and Koltun, 2013; Gu et al., 2016; Fu et al., 2015; Weinstein and Botvinick, 2017) has shown that a perfect global model is not necessary, and prior knowledge can be ﬁne-tuned to handle small changes.

Original: These methods, however, face a mismatch between what the model is trained for and how it is used at test time.
Revised: These methods, however, face a mismatch between what the model is trained for and how it is used at test time.

Original: In this paper, we bridge this gap by explicitly training a model for fast and effective adaptation.
Revised: In this paper, we bridge this gap by explicitly training a model for fast and effective adaptation.

Original: As a result, our model achieves more effective adaptation compared to these prior works, as validated in our experiments.
Revised: As a result, our model achieves more effective adaptation compared to these prior works, as validated in our experiments.

Original: Our problem setting relates to meta-learning, a long-standing problem of interest in machine learning that is concerned with enabling artiﬁcial agents to efﬁciently learn new tasks by learning to learn (Thrun and Pratt, 1998; Schmidhuber and Huber, 1991; Naik and Mammone, 1992; Lake et al., 2015).
Revised: Our problem setting relates to meta-learning, a long-standing problem of interest in machine learning that is concerned with enabling artiﬁcial agents to efﬁciently learn new tasks by learning to learn (Thrun and Pratt, 1998; Schmidhuber and Huber, 1991; Naik and Mammone, 1992; Lake et al., 2015).

Original: A meta-learner can control learning through approaches such as deciding the learner’s architecture (Baker et al., 2016), or by prescribing an optimization algorithm or update rule for the learner (Bengio et al., 1990; Schmidhuber, 1992; Younger et al., 2001; Andrychowicz et al., 2016; Li and Malik, 2016; Ravi and Larochelle, 2018).
Revised: A meta-learner can control learning through approaches such as deciding the learner’s architecture (Baker et al., 2016), or by prescribing an optimization algorithm or update rule for the learner (Bengio et al., 1990; Schmidhuber, 1992; Younger et al., 2001; Andrychowicz et al., 2016; Li and Malik, 2016; Ravi and Larochelle, 2018).

Original: Another popular meta-learning approach involves simply unrolling a recurrent neural network (RNN) that ingests the data (Santoro et al., 2016; Munkhdalai and Yu, 2017; Munkhdalai et al., 2017; Mishra et al., 2017) and learns internal representations of the algorithms themselves.
Revised: Another popular meta-learning approach involves simply unrolling a recurrent neural network (RNN) that ingests the data (Santoro et al., 2016; Munkhdalai and Yu, 2017; Munkhdalai et al., 2017; Mishra et al., 2017) and learns internal representations of the algorithms themselves.

Original: In particular, Duan et al.
Revised: In particular, Duan et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: Sung et al.
Revised: Sung et al.

Original: ; Al-Shedivat et al.
Revised: ; Al-Shedivat et al.

Original: (2017) developed model-free meta reinforcement learning methods.
Revised: (2017) developed model-free meta reinforcement learning methods.

Original:  In this work, we propose a model-based meta RL approach for learning online adaptation, suggest two instantiations of this general approach, and show resultsof our approach on  simulated agents as well as a real legged robot.To our knowledge, no prior work has developed a model-based meta reinforcement learning algorithm.
Revised: Recent work (Sæmundsson et al., 2018) has also framed meta-learning as a hierarchical latent variable model and applied it in a model-based RL; the modeling is done with GPs, adaptation is done after a trial, and results are shown on the tasks of cart-pole and double-pendulum swing-up. In contrast, we propose an approach for learning online adaptation of high-capacity neural network dynamics models; we present two instantiations of this general approach and show results on both simulated agents and a real legged robot.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: In this section, we present model-based reinforcement learning, and introduce the meta-learning formulation, where we describe the two main meta-learning approaches.
Revised: In this section, we present model-based reinforcement learning, and introduce the meta-learning formulation, where we describe the two main meta-learning approaches.

Original: 3. M ODEL -B ASED R EINFORCEMENT L EARNING
Revised: 3. M ODEL -B ASED R EINFORCEMENT L EARNING

Original: Reinforcement learning agents aim to perform actions that maximize some notion of cumulative reward.
Revised: Reinforcement learning agents aim to perform actions that maximize some notion of cumulative reward.

Original: Concretely, consider a Markov decision process (MDP) deﬁned by the tuple ( S , A , p, r, γ, ρ 0 , H ) .
Revised: Concretely, consider a Markov decision process (MDP) deﬁned by the tuple ( S , A , p, r, γ, ρ 0 , H ) .

Original: Here, S is the set of states, A is the set of actions, p ( s (cid:48) | s , a ) is the state transition distribution, r : S ×
Revised: Here, S is the set of states, A is the set of actions, p ( s (cid:48) | s , a ) is the state transition distribution, r : S ×

Original: A → R is a bounded reward function, ρ 0 : S → R + is the initial state distribution, γ is the discount factor, and H is the horizon.
Revised: A → R is a bounded reward function, ρ 0 : S → R + is the initial state distribution, γ is the discount factor, and H is the horizon.

Original: A trajectory segment is denoted by τ ( i, j ) :
Revised: A trajectory segment is denoted by τ ( i, j ) :

Original: = ( s i , a i , ..., s j , a j , s j +1 ) .
Revised: = ( s i , a i , ..., s j , a j , s j +1 ) .

Original: Finally, the sum of expected rewards from a trajectory is the return.
Revised: Finally, the sum of expected rewards from a trajectory is the return.

Original: In this framework, RL aims to ﬁnd a policy π : S
Revised: In this framework, RL aims to ﬁnd a policy π : S

Original: →
Revised: →

Original: A that prescribes the optimal action to take from each state in order to maximize the expected return.
Revised: A that prescribes the optimal action to take from each state in order to maximize the expected return.

Original: Model-based RL aims to solve this problem by learning the transition distribution p ( s (cid:48) | s , a ) , which is also referred to as the dynamics model.
Revised: Model-based RL aims to solve this problem by learning the transition distribution p ( s (cid:48) | s , a ) , which is also referred to as the dynamics model.

Original: This can be done using a function approximator ˆ p θ ( s (cid:48) | s , a ) to approximate the dynamics, where the weights θ are optimized to maximize the log-likelihood of the observed data D .
Revised: This can be done using a function approximator ˆ p θ ( s (cid:48) | s , a ) to approximate the dynamics, where the weights θ are optimized to maximize the log-likelihood of the observed data D .

Original: In practice, this model is then used in the process of action selection by either producing data points from which to train a policy, or by producing predictions and dynamics constraints to be optimized by a controller.
Revised: In practice, this model is then used in the process of action selection by either producing data points from which to train a policy, or by producing predictions and dynamics constraints to be optimized by a controller.

Original: 3. M ETA -L EARNING
Revised: 3. M ETA -L EARNING

Original: Meta-learning is concerned with automatically learning learning algorithms that are more efﬁcient and effective than learning from scratch.
Revised: Meta-learning is concerned with automatically learning learning algorithms that are more efﬁcient and effective than learning from scratch.

Original: These algorithms leverage data from previous tasks to acquire a learning procedure that can quickly adapt to new tasks.
Revised: These algorithms leverage data from previous tasks to acquire a learning procedure that can quickly adapt to new tasks.

Original: These methods operate under the assumption that the previous meta-training tasks and the new meta-test tasks are drawn from the same task distribution ρ ( T ) and share a common structure that can be exploited for fast learning.
Revised: These methods operate under the assumption that the previous meta-training tasks and the new meta-test tasks are drawn from the same task distribution ρ ( T ) and share a common structure that can be exploited for fast learning.

Original: In the supervised learning setting, we aim to learn a function f
Revised: In the supervised learning setting, we aim to learn a function f

Original: θ with parameters θ that minimizes a supervised loss L T .
Revised: θ with parameters θ that minimizes a supervised loss L T .

Original: Then, the goal of meta-learning is to ﬁnd a learning procedure, denoted as θ (cid:48)
Revised: Then, the goal of meta-learning is to ﬁnd a learning procedure, denoted as θ (cid:48)

Original: =
Revised: =

Original: u ψ ( D tr T , θ ) , that can learn a range of tasks T from small datasets D tr T .
Revised: u ψ ( D tr T , θ ) , that can learn a range of tasks T from small datasets D tr T .

Original: We can formalize this meta-learning problem setting as optimizing for the parameters of the learning procedure θ , ψ as follows:
Revised: We can formalize this meta-learning problem setting as optimizing for the parameters of the learning procedure θ , ψ as follows:

Original: [Equation]
Revised: [Equation]

Original: where D tr T , D test T are sampled without replacement from the meta-training dataset D T .
Revised: where D tr T , D test T are sampled without replacement from the meta-training dataset D T .

Original: Once meta-training optimizes for the parameters θ ∗ , ψ ∗ , the learning procedure u ψ ( · , θ ) can then be used to learn new held-out tasks from small amounts of data.
Revised: Once meta-training optimizes for the parameters θ ∗ , ψ ∗ , the learning procedure u ψ ( · , θ ) can then be used to learn new held-out tasks from small amounts of data.

Original: We will also refer to the learning procedure u as the update function.
Revised: We will also refer to the learning procedure u as the update function.

Original: Gradient-based meta-learning.
Revised: Gradient-based meta-learning.

Original: Model-agnostic meta-learning (MAML)
Revised: Model-agnostic meta-learning (MAML)

Original: (Finn et al., 2017) aims to learn the initial parameters of a neural network such that taking one or several gradient descent steps from this initialization leads to effective generalization (or few-shot generalization) to new tasks.
Revised: (Finn et al., 2017) aims to learn the initial parameters of a neural network such that taking one or several gradient descent steps from this initialization leads to effective generalization (or few-shot generalization) to new tasks.

Original: Then, when presented with new tasks, the model with the meta-learned initialization can be quickly ﬁne-tuned using a few data points from the new tasks.
Revised: Then, when presented with new tasks, the model with the meta-learned initialization can be quickly ﬁne-tuned using a few data points from the new tasks.

Original: Using the notation from before, MAML uses gradient descent as a learning algorithm:
Revised: Using the notation from before, MAML uses gradient descent as a learning algorithm:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The learning rate α may be a learnable paramter (in which case ψ = α ) or ﬁxed as a hyperparameter, leading to ψ = ∅ .
Revised: The learning rate α may be a learnable paramter (in which case ψ = α ) or ﬁxed as a hyperparameter, leading to ψ = ∅ .

Original: Despite the update rule being ﬁxed, a learned initialization of an overparameterized deep network followed by gradient descent is as expressive as update rules represented by deep recurrent networks (Finn and Levine, 2017).
Revised: Despite the update rule being ﬁxed, a learned initialization of an overparameterized deep network followed by gradient descent is as expressive as update rules represented by deep recurrent networks (Finn and Levine, 2017).

Original: Recurrence-based meta-learning.
Revised: Recurrence-based meta-learning.

Original: Another approach to meta-learning is to use recurrent models.
Revised: Another approach to meta-learning is to use recurrent models.

Original: In this case, the update function is always learned, and ψ corresponds to the weights of the recurrent model that update the hidden state.
Revised: In this case, the update function is always learned, and ψ corresponds to the weights of the recurrent model that update the hidden state.

Original: The parameters θ of the prediction model correspond to the remainder of the weights of the recurrent model and the hidden state.
Revised: The parameters θ of the prediction model correspond to the remainder of the weights of the recurrent model and the hidden state.

Original: Both gradient-based and recurrence-based meta-learning methods have been used for meta model-free RL (Finn et al., 2017;
Revised: Both gradient-based and recurrence-based meta-learning methods have been used for meta model-free RL (Finn et al., 2017;

Original: Duan et al., 2016).
Revised: Duan et al., 2016).

Original: We will build upon these ideas to develop a meta model-based RL algorithm that enables adaptation in dynamic environments, in an online way.
Revised: We will build upon these ideas to develop a meta model-based RL algorithm that enables adaptation in dynamic environments, in an online way.

Original:  M ETA -L EARNING FOR O NLINE M ODEL A DAPTATION
Revised:  M ETA -L EARNING FOR O NLINE M ODEL A DAPTATION

Original: In this section, we present our approach for meta-learning for online model adaptation.
Revised: In this section, we present our approach for meta-learning for online model adaptation.

Original: As explained in Section 3.2, standard meta-learning formulations require the learned model θ ∗ , ψ ∗ to adapt after seeing M data points from some new “task.”
Revised: As explained in Section 3.2, standard meta-learning formulations require the learned model θ ∗ , ψ ∗ to adapt after seeing M data points from some new “task.”

Original: Our notion of task is slightly more ﬂuid, where every segment of a trajectory can be considered to be a different “task,” and the past M observations can be considered as providing more information about the current task setting.
Revised: Our notion of task is slightly more ﬂuid, where every segment of a trajectory can be considered to be a different “task,” and the past M observations can be considered as providing more information about the current task setting.

Original: Since changes in system dynamics, terrain details, or other environmental changes can occur at any time, we consider (at every time step) the problem of adapting to the M past time steps.
Revised: Since changes in system dynamics, terrain details, or other environmental changes can occur at any time, we consider (at every time step) the problem of adapting to the M past time steps.

Original: In this work, we use the notion of environment E to denote different settings or conﬁgurations of a particular problem, ranging from malfunction in the joints of a robot to the state of external disturbances.
Revised: In this work, we use the notion of environment E to denote different settings or conﬁgurations of a particular problem, ranging from malfunction in the joints of a robot to the state of external disturbances.

Original: We assume a distribution of environments ρ ( E ) that share some common structure, such as the same observation and action space, but may differ in their dynamics p E ( s (cid:48) | s , a ) .
Revised: We assume a distribution of environments ρ ( E ) that share some common structure, such as the same observation and action space, but may differ in their dynamics p E ( s (cid:48) | s , a ) .

Original: We denote a trajectory by τ E ( i, j ) , which represents a sequence of states and actions ( s i , a i , ..., s j , a j , s j +1 ) sampled from environment E .
Revised: We denote a trajectory by τ E ( i, j ) , which represents a sequence of states and actions ( s i , a i , ..., s j , a j , s j +1 ) sampled from environment E .

Original: We pose the meta-RL problem in this setting as an optimization over ( θ , ψ ) with respect to a maximum likelihood meta-objective.
Revised: We pose the meta-RL problem in this setting as an optimization over ( θ , ψ ) with respect to a maximum likelihood meta-objective.

Original: The meta-objective is the likelihood of the data under a predictive model ˆ p θ (cid:48)
Revised: The meta-objective is the likelihood of the data under a predictive model ˆ p θ (cid:48)

Original: ( s (cid:48) | s , a ) with parameters θ (cid:48) , where θ (cid:48)
Revised: ( s (cid:48) | s , a ) with parameters θ (cid:48) , where θ (cid:48)

Original: =
Revised: =

Original: u ψ ( τ E ( t − M, t − 1) , θ ) corresponds to model parameters that were updated using the past M data points.
Revised: u ψ ( τ E ( t − M, t − 1) , θ ) corresponds to model parameters that were updated using the past M data points.

Original: Concretely, this corresponds to the following optimization:
Revised: Concretely, this corresponds to the following optimization:

Original: [Equation]
Revised: [Equation]

Original: where the loss L corresponds to the negative log likelihood of the data under the model:
Revised: where the loss L corresponds to the negative log likelihood of the data under the model:

Original: [Equation]
Revised: [Equation]

Original: Require: Distribution ρ E over tasks

Original: Require:

Original: Learning rate β ∈ R +

Original: Require:

Original: Number of sampled tasks N , dataset D

Original: Require:

Original: Task sampling frequency n S ∈ Z + 1: Randomly initialize θ 2: for i = 1 , ... do 3: if i mod n S = 0 then 4:

Original: Sample E ∼ ρ ( E ) 5:

Original: Collect τ E using Alg.6:

Original: D ← D ∪ { τ E } 7: end if 8: for j = 1 . . .

Original: N do 9: τ E ( t − M, t − 1) , τ E ( t, t + K ) ∼ D 10: θ (cid:48)E ← u ψ ( τ E ( t − M, t − 1) , θ ) 11:

Original: L j ← L ( τ E ( t, t + K ) , θ (cid:48)E ) 12: end for 13: θ ← θ − β ∇ θ 1 N N (cid:80)

Original: j =1 L j 14: ψ ← ψ − η ∇ ψ 1 N N (cid:80) j

Original: =1 L j 15: end for 16: Return ( θ , ψ ) as ( θ ∗ , ψ ∗ )

Original: Algorithm

Original: (test time)

Original: Require: Meta-learned parameters θ ∗ , ψ ∗

Original: Require: controller () , H , r , n A 1: D ← ∅ 2: for each timestep t do 3: θ (cid:48)∗ ← u ψ ∗ ( D ( t − M, t − 1) , θ ∗ ) 4: a ← controller ( θ (cid:48)∗ , r, H, n A ) 5:

Original: Execute a , add result to D 6: end for 7: Return rollout D

Original: In the meta-objective in Equation 3, note that the past M points are used to adapt θ into θ (cid:48) , and the loss of this θ (cid:48) is evaluated on the future K points.
Revised: In the meta-objective in Equation 3, note that the past M points are used to adapt θ into θ (cid:48) , and the loss of this θ (cid:48) is evaluated on the future K points.

Original: Thus, we use the past M timesteps to provide insight into how to adapt our model to perform well for nearby future timesteps.
Revised: Thus, we use the past M timesteps to provide insight into how to adapt our model to perform well for nearby future timesteps.

Original: As outlined in
Revised: As outlined in

Original: Algorithm 1, the update rule u ψ for the inner update and a gradient step on θ for the outer update allow us to optimize this meta-objective of adaptation.
Revised: Algorithm 1, the update rule u ψ for the inner update and a gradient step on θ for the outer update allow us to optimize this meta-objective of adaptation.

Original: Thus, we achieve fast adaptation at test time by being able to ﬁne-tune the model using just M data points.
Revised: Thus, we achieve fast adaptation at test time by being able to ﬁne-tune the model using just M data points.

Original: While we focus on reinforcement learning problems in our experiments, this meta-learning approach could be used for a learning to adapt online in a variety of sequence modeling domains.
Revised: While we focus on reinforcement learning problems in our experiments, this meta-learning approach could be used for a learning to adapt online in a variety of sequence modeling domains.

Original: We present our algorithm using both a recurrence and a gradient-based meta-learner, as we discuss next.
Revised: We present our algorithm using both a recurrence and a gradient-based meta-learner, as we discuss next.

Original: Gradient-Based Adaptive Learner (GrBAL).
Revised: Gradient-Based Adaptive Learner (GrBAL).

Original: GrBAL uses a gradient-based meta-learning to perform online adaptation; in particular, we use MAML (Finn et al., 2017).
Revised: GrBAL uses a gradient-based meta-learning to perform online adaptation; in particular, we use MAML (Finn et al., 2017).

Original: In this case, our update rule is prescribed by gradient descent ( 5.)
Revised: In this case, our update rule is prescribed by gradient descent ( 5.)

Original: [Equation]
Revised: [Equation]

Original: Recurrence-Based Adaptive Learner (ReBAL).
Revised: Recurrence-Based Adaptive Learner (ReBAL).

Original: ReBAL, instead, utilizes a recurrent model, which learns its own update rule (i.e., through its internal gating structure.
Revised: ReBAL, instead, utilizes a recurrent model, which learns its own update rule (i.e., through its internal gating structure.

Original: In this case, ψ and u ψ corresponds to the weights of the recurrent model that update its hidden state.
Revised: In this case, ψ and u ψ corresponds to the weights of the recurrent model that update its hidden state.

Original:  M ODEL -B ASED M ETA -R EINFORCEMENT L EARNING
Revised:  M ODEL -B ASED M ETA -R EINFORCEMENT L EARNING

Original: Now that we have discussed our approach for enabling online adaptation, we next propose how to build upon this idea to develop a model-based meta-reinforcement learning algorithm.
Revised: Now that we have discussed our approach for enabling online adaptation, we next propose how to build upon this idea to develop a model-based meta-reinforcement learning algorithm.

Original: First, we explain how the agent can use the adapted model to perform a task, given parameters θ ∗ and ψ ∗ from optimizing the meta-learning objective.
Revised: First, we explain how the agent can use the adapted model to perform a task, given parameters θ ∗ and ψ ∗ from optimizing the meta-learning objective.

Original: Given θ ∗ and ψ ∗ , we use the agent’s recent experience to adapt the model parameters: θ (cid:48)∗ = u ψ ∗ ( τ ( t − M, t ) , θ ∗ ) .
Revised: Given θ ∗ and ψ ∗ , we use the agent’s recent experience to adapt the model parameters: θ (cid:48)∗ = u ψ ∗ ( τ ( t − M, t ) , θ ∗ ) .

Original: This results in a model
Revised: This results in a model

Original: ˆ p θ (cid:48)∗ that better captures the local dynamics in the current setting, task, or environment.
Revised: ˆ p θ (cid:48)∗ that better captures the local dynamics in the current setting, task, or environment.

Original: This adapted model is then passed to our controller, along with the reward function r and a planning horizon H .
Revised: This adapted model is then passed to our controller, along with the reward function r and a planning horizon H .

Original: We use a planning H that is smaller than the adaptation horizon K , since the adapted model is only valid within the current context.
Revised: We use a planning H that is smaller than the adaptation horizon K , since the adapted model is only valid within the current context.

Original: We use model predictive path integral control (MPPI)
Revised: We use model predictive path integral control (MPPI)

Original: (Williams et al., 2015), but, in principle, our model adaptation approach is agnostic to the model predictive control (MPC) method used.
Revised: (Williams et al., 2015), but, in principle, our model adaptation approach is agnostic to the model predictive control (MPC) method used.

Original: The use of MPC compensates for model inaccuracies by preventing accumulating errors, since we replan at each time step using updated state information.
Revised: The use of MPC compensates for model inaccuracies by preventing accumulating errors, since we replan at each time step using updated state information.

Original: MPC also allows for further beneﬁts in this setting of online adaptation, because the model ˆ p θ (cid:48)E itself will also improve by the next time step.
Revised: MPC also allows for further beneﬁts in this setting of online adaptation, because the model ˆ p θ (cid:48)E itself will also improve by the next time step.

Original: After taking each step, we append the resulting state transition onto our dataset, reset the model parameters back to θ ∗ , and repeat the entire planning process for each timestep.
Revised: After taking each step, we append the resulting state transition onto our dataset, reset the model parameters back to θ ∗ , and repeat the entire planning process for each timestep.

Original: See Algorithm 2 for this adaptation procedure.
Revised: See Algorithm 2 for this adaptation procedure.

Original: Finally, in addition to test-time, we also perform this online adaptation procedure during the meta-training phase itself, to provide on-policy rollouts for meta-training.
Revised: Finally, in addition to test-time, we also perform this online adaptation procedure during the meta-training phase itself, to provide on-policy rollouts for meta-training.

Original: For the complete meta-RL algorithm, see Algorithm 1.
Revised: For the complete meta-RL algorithm, see Algorithm 1.

Original:  R ESULTS
Revised:  R ESULTS

Original: Our evaluation aims to answer the following questions: (1) is adaptation actually changing the model?
Revised: Our evaluation aims to answer the following questions: (1) is adaptation actually changing the model?

Original: , (2) does our approach enable fast adaptation to varying dynamics, tasks, and environments, both inside and outside of the training distribution?
Revised: , (2) does our approach enable fast adaptation to varying dynamics, tasks, and environments, both inside and outside of the training distribution?

Original: , (3) how does our method’s performance compare to that of other methods?, (4) how does GrBAL and ReBAL compare?, (6) how does meta model-based
Revised: , (3) how does our method’s performance compare to Require: Distribution ρ E over tasks Require: Learning rate β ∈ R +

Revised: Require:

Original: RL compare to meta model-free RL in terms of sample efﬁciency and performance for these tasks?
Revised: Number of sampled tasks N , dataset D Require: Task sampling frequency n S ∈ Z + 1: Randomly initialize θ 2: for i = 1 , ... do 3: if i mod n S = 0 then 4: Sample E ∼ ρ ( E ) 5:

Revised: Collect τ E using Alg.6:

Revised: D ← D ∪ { τ E } 7: end if 8: for j = 1 . . .

Revised: N do 9: τ E ( t − M, t − 1) , τ E ( t, t + K ) ∼ D 10: θ (cid:48)E ← u ψ ( τ E ( t − M, t − 1) , θ ) 11:

Revised: L j ← L ( τ E ( t, t + K ) , θ (cid:48)E ) 12: end for 13: θ ← θ − β ∇ θ 1 N N (cid:80)

Revised: j =1 L j 14: ψ ← ψ − η ∇ ψ 1 N N (cid:80) j

Revised: =1 L j 15: end for 16: Return ( θ , ψ ) as ( θ ∗ , ψ ∗ )

Revised: Algorithm

Revised: (test time)

Revised: Require: Meta-learned parameters θ ∗ , ψ ∗

Revised: Require: controller () , H , r , n A 1: D ← ∅ 2: for each timestep t do 3: θ (cid:48)∗ ← u ψ ∗ ( D ( t − M, t − 1) , θ ∗ ) 4: a ← controller ( θ (cid:48)∗ , r, H, n A ) 5:

Revised: Execute a , add result to D 6: end for 7: Return rollout D that of other methods?, (4) how does GrBAL and ReBAL compare?

Revised: , (5) how does meta model-based

Revised: RL compare to meta model-free RL in terms of sample efﬁciency and performance for these tasks?

Original: , and (7) can our method learn to adapt online on a real robot, and if so, how do it perform?
Revised: , and (6) can our method learn to adapt online on a real robot, and if so, how do it perform?

Original: We next present our set-up and results, motivated by these questions.
Revised: We next present our set-up and results, motivated by these questions.

Original: Videos of all results are online.
Revised: Videos of all results are online.

Original: We ﬁrst conduct a comparative evaluation of our algorithm, on a variety of simulated robots using the
Revised: We ﬁrst conduct a comparative evaluation of our algorithm, on a variety of simulated robots using the

Original: MuJoCo physics engine (Todorov et al., 2012).
Revised: MuJoCo physics engine (Todorov et al., 2012).

Original: For all of our environments, we model the transition probabilities as Gaussian random variables with mean parameterized by the neural network model, and ﬁxed variance.
Revised: For all of our environments, we model the transition probabilities as Gaussian random variables with mean parameterized by the neural network model, and ﬁxed variance.

Original: In this case, maximum likelihood estimation corresponds to minimizing the mean squared error.
Revised: In this case, maximum likelihood estimation corresponds to minimizing the mean squared error.

Original: We now describe the setup of our environments (Fig. 2), where each agent requires different types of adaptation:
Revised: We now describe the setup of our environments (Fig. 2), where each agent requires different types of adaptation:

Original: Half-cheetah (HC): disabled joint.
Revised: Half-cheetah (HC): disabled joint.

Original: For each rollout during meta-training, we randomly sample a joint to be disabled (i.e., the agent cannot apply torques to that joint).
Revised: For each rollout during meta-training, we randomly sample a joint to be disabled (i e., the agent cannot apply torques to that joint).

Original: HC: sloped terrain.
Revised: HC: sloped terrain.

Original: For each rollout during meta-training, we randomly select an upward or downward slope of low steepness.
Revised: For each rollout during meta-training, we randomly select an upward or downward slope of low steepness.

Original: HC: pier.
Revised: HC: pier.

Original: In this task, the cheetah runs over a series of blocks that are ﬂoating on water.
Revised: In this task, the cheetah runs over a series of blocks that are ﬂoating on water.

Original: Each block moves up and down when stepped on, and the changes in the dynamics are rapidly changing, due to each block having different damping and friction properties.
Revised: Each block moves up and down when stepped on, and the changes in the dynamics are rapidly changing, due to each block having different damping and friction properties.

Original: The HC is meta-trained by varying these block properties, and tested on a speciﬁc (randomly-selected) conﬁguration of block properties.
Revised: The HC is meta-trained by varying these block properties, and tested on a speciﬁc (randomly-selected) conﬁguration of block properties.

Original: Ant: crippled leg.
Revised: Ant: crippled leg.

Original: For each meta-training rollout, we randomly sample a leg to cripple on this quadrupedal robot.
Revised: For each meta-training rollout, we randomly sample a leg to cripple on this quadrupedal robot.

Original: This causes unexpected and drastic changes to the underlying dynamics.
Revised: This causes unexpected and drastic changes to the underlying dynamics.

Original: 6. E FFECT OF A DAPTATION
Revised: 6. E FFECT OF A DAPTATION

Original: First, we analyze the effect of the model adaptation, and show results from three environments: HC pier, HC sloped terrain running up and down a steep hill, and ant crippled leg, where the particular crippled leg is not seen during training.
Revised: First, we analyze the effect of the model adaptation, and show results from three environments: HC pier, HC sloped terrain running up and down a steep hill, and ant crippled leg, where the particular crippled leg is not seen during training.

Original: Figuredisplays the distribution shift between the pre-update and post-update model prediction errors of three GrBAL runs, showing that using the past M timesteps to update θ ∗ (pre) into θ (cid:48)∗ (post) does indeed reduce model error on predicting the following K timesteps.
Revised: Figure 3 displays the distribution shift between the preupdate and post-update model prediction errors of three GrBAL runs, showing that using the past M timesteps to update θ ∗ (pre) into θ (cid:48)∗ (post) does indeed reduce model error on predicting the following K timesteps.

Original: 6. S AMPLE E FFICIENCY
Revised: 6. S AMPLE E FFICIENCY

Original: For our ﬁrst comparative evaluation, we evaluate both GrBAL and ReBAL against a state-of-theart model-free meta-RL method (MAML-RL)
Revised: For our ﬁrst comparative evaluation, we evaluate both GrBAL and ReBAL against a state-of-the-art model-free meta-RL method (MAML-RL)

Original: Finn et al. (2017) and a purely model-free method (TRPO) Schulman et al.
Revised: (Finn et al., 2017) to showcase the beneﬁt of model-based approaches, and a purely model-free method (TRPO) (Schulman et al., 2015) to showcase the beneﬁt of meta-learning.

Original: Here, we train both model-free approaches until convergence, and they receive the equivalent of about two days of real-world experience.
Revised: We train both methods until convergence, using the equivalent of about two days of real-world experience.

Original: Our model-based meta-RL approach results in superior or equivalent performance than a model-free agent trained withtimes more data.
Revised: Our model-based meta-RL approach results in superior or equivalent performance to a model-free agent trained with 1000 times more data.

Original: When comparing it against a model-free meta-RL method, our approach asymptotic performance falls behind. However, it is able to accomplish the task with suboptimal performance requiring, again, the equivalent of 1000 times less data.
Revised: When comparing it against a model-free meta-RL method, our approach’s asymptotic performance falls behind, but achieves this suboptimal performance with, again, the equivalent of 1000 times less data.

Original: In the case of half-cheetah disabled, the performancereached closely matches the asymptotic performance of the model-free meta-RL method.
Revised: In the case of half-cheetah disabled, our performance closely matches the asymptotic performance of the model-free meta-RL method.

Original: 6. F AST A DAPTATION & G ENERALIZATION
Revised: 6. F AST A DAPTATION & G ENERALIZATION

Original: For our second comparative evaluation, we evaluate both GrBAL and ReBAL against 1) a non-adaptive model-based method (“MB”), which employs a feedforward neural network as the dynamics model and selects actions using MPC, but does not perform adaptation (Nagabandi et al., 2017a), 2) an adaptive control method (“MB + DE") based on dynamic-evaluation, where gradient steps are taken to adapt a model at run time (Krause et al., 2017), and 3) 
Revised: For our second comparative evaluation, we evaluate both GrBAL and ReBAL against 1) a nonadaptive model-based method (“MB”), which employs a feedforward neural network as the dynamics model and selects actions using MPC (Nagabandi et al., 2017a), 2) an adaptive control method (“MB + DE") based on dynamic-evaluation, where gradient steps are taken to adapt a model at run time (Krause et al., 2017), and 3) a model-free method (“TRPO")

Original: TRPO (Schulman et al., 2015).
Revised: (Schulman et al., 2015).

Revised: Note that for meta-learning with adaptation (ours), adaptation (MB+DE), and none (MB), we use models and techniques from these cited works; however, we choose to employ the same choice of MPC controller to allow for fair comparisons (i.e., MPPI for simulated agents with higher action dimensions, and random-shooting for the real agent with low action dimensions).

Original: Inthis comparison, we operate in the low data regime, giving all of these methods the same amount of data, in the interest of developing efﬁcient algorithms for real-world applications.
Revised: In the interest of developing efﬁcient algorithms for real-world applications, we operate in the low data regime for all experiments in this comparison.

Original: This translates to  1.5-3 hours of real-world experience.
Revised: This translates to only 1.5-3 hours of real-world experience.

Original: We also provide the performance of a MB oracle, which is trained using unlimiteddata, and using only data from  the given test task (rather than being trained on the training tasks and being required to generalize).
Revised: We also provide the performance of a MB oracle, which is trained using unlimited data from only the given test task (rather than needing to generalize to various training tasks).

Original: In these experiments, we test the ability of each approach to adapt to sudden changes in the environment, as well as to generalize beyond the training environments.
Revised: In these experiments, we test the ability of each approach to adapt to sudden changes in the environment, as well as to generalize beyond the training environments.

Original: We evaluate the fast adaptation (F.A.) component on the HC disabled joint, ant crippled leg, and the HC pier.
Revised: We evaluate the fast adaptation (F.A.) component on the HC disabled joint, ant crippled leg, and the HC pier.

Original: On the ﬁrst two, we cause a joint/leg of the robot to malfunction in the middle of a rollout.
Revised: On the ﬁrst two, we cause a joint/leg of the robot to malfunction in the middle of a rollout.

Original: We evaluate the generalization  component also on the tasks of HC disabled joint and ant crippled leg, but this time, the leg/joint that malfunctions has not been seen as crippled during training.
Revised: We evaluate the generalization [Table] component also on the tasks of HC disabled joint and ant crippled leg, but this time, the leg/joint that malfunctions has not been seen as crippled during training.

Original: The last environment that we test generalization on is the HC slopped terrain for a hill, where the agent has to run up and down a steep slope, which is outside of the gentle slopes that it experienced during training.
Revised: The last environment that we test generalization on is the HC slopped terrain for a hill, where the agent has to run up and down a steep slope, which is outside of the gentle slopes that it experienced during training.

Original: The results, shown inFig.
Revised: The results, shown in

Original:  5, show returns that are normalized such that the MB oracle achieves a return of 1.
Revised: Fig. 5, show returns that are normalized such that the MB oracle achieves a return of 1.

Original: In all tasks, due to low quantity of training data, TRPO performs poorly.
Revised: In all tasks, due to low quantity of training data, TRPO performs poorly.

Original: Although MB+DE achieves better generalization than MB, the slow nature of its adaptation causes it to fall behind MB in the[Table] environments that require fast adaptation.
Revised: Although MB+DE achieves better generalization than MB, the slow nature of its adaptation causes it to fall behind MB in the environments that require fast adaptation.

Original: On the other hand, our approach surpasses the other approaches in all of the tasks.
Revised: On the other hand, our approach surpasses the other approaches in all of the tasks.

Original: In fact, in the HC pier and the fast adaptation of ant environments, our approach surpasses the model-based oracle.
Revised: In fact, in the HC pier and the fast adaptation of ant environments, our approach surpasses the model-based oracle.

Original: This result showcases the importance of adaptation in stochastic environments where even a model trained with a lot of data cannot be robust to unexpected occurrences or disturbances.
Revised: This result showcases the importance of adaptation in stochastic environments, where even a model trained with a lot of data cannot be robust to unexpected occurrences or disturbances.

Original: ReBAL displays its strengths on tasks where longer sequential inputs allow it to better asses current task settings but overall, GrBAL seems to perform better for both of these cases of generalization and fast adaptation.
Revised: ReBAL displays its strengths on tasks where longer sequential inputs allow it to better asses current task settings, but overall, GrBAL seems to perform better for both of these cases of generalization and fast adaptation.

Original: 6. R EAL -W ORLD R ESULTS
Revised: 6. R EAL -W ORLD R ESULTS

Original: To test our meta model-based RL method’s sample efﬁciency, as well as its ability to perform fast and effective online adaptation, we applied GrBAL to a real legged millirobot.
Revised: To test our meta model-based RL method’s sample efﬁciency, as well as its ability to perform fast and effective online adaptation, we applied GrBAL to a real legged millirobot.

Original: Due to the cost of running real robot experiments, we choose the better performing method (i e., GrBAL) to evaluate on the real robot.
Revised: Due to the cost of running real robot experiments, we choose the better performing method (i.e., GrBAL) to evaluate on the real robot.

Original: This small 6-legged robot, as shown in Fig.
Revised: This small 6-legged robot, as shown in Fig.

Original: 1 and Fig. 2, presents a modeling and control challenge in the form of highly stochastic and dynamic movement.
Revised: 1 and Fig. 2, presents a modeling and control challenge in the form of highly stochastic and dynamic movement.

Original: This robot is an excellent candidate for online adaptation for many reasons: the rapid manufacturing techniques and numerous custom-design steps used to construct this robot make it impossible to reproduce the same dynamics each time, its linkages and other body parts deteriorate over time, and it moves very quickly and dynamically with bounding-style gaits; hence, its dynamics are strongly dependent on the terrain or task at hand.
Revised: This robot is an excellent candidate for online adaptation for many reasons: the rapid manufacturing techniques and numerous custom-design steps used to construct this robot make it impossible to reproduce the same dynamics each time, its linkages and other body parts deteriorate over time, and it moves very quickly and dynamically with bounding-style gaits; hence, its dynamics are strongly dependent on the terrain or task at hand.

Original: The state space of the robot is a 24-dimensional vector, including center of mass positions and velocities, center of mass pose and angular velocities, back-EMF readings of motors, encoder readings of leg motor angles and velocities, and battery voltage.
Revised: The state space of the robot is a 24-dimensional vector, including center of mass positions and velocities, center of mass pose and angular velocities, back-EMF readings of motors, encoder readings of leg motor angles and velocities, and battery voltage.

Original: We deﬁne the action space to be velocity setpoints of the rotating legs.
Revised: We deﬁne the action space to be velocity setpoints of the rotating legs.

Original: The action space has a dimension of two, since one motor on each side is coupled to all three of the legs on that side.
Revised: The action space has a dimension of two, since one motor on each side is coupled to all three of the legs on that side.

Original: All experiments are conducted in a motion capture room.
Revised: All experiments are conducted in a motion capture room.

Original: Computation is done on an external computer, and the velocity setpoints are streamed over radio at 10 Hz to be executed bya PID controller on the microcontroller on-board of the robot.
Revised: Computation is done on an external computer, and the velocity setpoints are streamed over radio at 10 Hz to be executed by

Original: We meta-train a dynamics model for this robot using the meta-objective described in Equation 3, and we train it to adapt on entirely real-world data from three different training terrains: carpet, styrofoam, and turf.
Revised: We meta-train a dynamics model for this robot using the meta-objective described in Equation 3, and we train it to adapt on entirely real-world data from three different training terrains: carpet, styrofoam, and turf.

Original: We collect approximately 30 minutes of data from each of the three training terrains.
Revised: We collect approximately 30 minutes of data from each of the three training terrains.

Original: This data was entirely collected using a random policy, in conjunction with a safety policy, whose sole purpose was to prevent the robot from exiting the area of interest.
Revised: This data was entirely collected using a random policy, in conjunction with a safety policy, whose sole purpose was to prevent the robot from exiting the area of interest.

Original: Our ﬁrst group of results (Table 1) show that, when data from a random policy is used to train a dynamics model, both a model trained with a standard supervised learning objective (MB) and
Revised: Our ﬁrst group of results (Table 1) show that, when data from a random policy is used to train a dynamics model, both a model trained with a standard supervised learning objective (MB) and

Original: a
Revised: a

Original: GrBAL model achieve comparable performance for executing desired trajectories on terrains from the training distribution.
Revised: GrBAL model achieve comparable performance for executing desired trajectories on terrains from the training distribution.

Original: Next, we test the performance of our method on what it is intended for: fast online adaptation of the learned model to enable successful execution of new, changing, or out-of-distribution tasks at test time.
Revised: Next, we test the performance of our method on what it is intended for: fast online adaptation of the learned model to enable successful execution of new, changing, or out-of-distribution tasks at test time.

Original: Similar to the comparisons above, we compare GrBAL to a model-based method (MB) that involves neither meta-training nor online adaptation, as well as a dynamic evaluation method that involves online adaptation of that MB model (MB+DE).
Revised: Similar to the comparisons above, we compare GrBAL to a model-based method (MB) that involves neither meta-training nor online adaptation, as well as a dynamic evaluation method that involves online adaptation of that MB model (MB+DE).

Original: Our results (Fig. 6) demonstrate that
Revised: Our results (Fig. 6) demonstrate that

Original: GrBAL substantially outperforms MB and MB+DE, and, unlike MB and MB+DE, and that GrBAL can quickly 1) adapt online to a missing leg, 2) adjust to novel terrains and slopes, 3) account for miscalibration or errors in pose estimation, and 4) compensate for pulling payloads.
Revised: GrBAL substantially outperforms MB and MB+DE, and, unlike MB and MB+DE, and that GrBAL can quickly 1) adapt online to a missing leg, 2) adjust to novel terrains and slopes, 3) account for miscalibration or errors in pose estimation, and 4) compensate for pulling payloads.

Original: None of these environments were seen during training time, but the agent’s ability to learn how to learn enables it to quickly adapt its prior knowledge and ﬁne-tune to learn new tasks online.
Revised: None of these environments were seen during training time, but the agent’s ability to learn how to learn enables it to quickly adapt its prior knowledge and ﬁne-tune to learn new tasks online.

Original: Furthermore, the poor performance of the MB and MB+DE baselines demonstrate not only the need for adaptation, but also the importance of good initial parameters to adapt from (in this case, meta-learned parameters).
Revised: Furthermore, the poor performance of the MB and MB+DE baselines demonstrate not only the need for adaptation, but also the importance of good initial parameters to adapt from (in this case, meta-learned parameters).

Original: The qualitative results of these experiments in Fig.
Revised: The qualitative results of these experiments in Fig.

Original: 7 show that the robot is able to use our method to adapt online and effectively follow the target trajectories, even in the presence of new tasks and unexpected perturbations at test time.
Revised: 7 show that the robot is able to use our method to adapt online and effectively follow the target trajectories, even in the presence of new tasks and unexpected perturbations at test time.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we present an approach for model-based meta reinforcement learning that enables fast, online adaptation in dynamic environments.
Revised: In this work we present an approach for model-based meta reinforcement learning that enables fast, online adaptation in dynamic environments.

Original: We show that meta-learning a model for online adaptation results in a method that is able to adapt to unseen situations or sudden and drastic changes in the environment, and is also sample efﬁcient to train.
Revised: We show that meta-learning a model for online adaptation results in a method that is able to adapt to unseen situations or sudden and drastic changes in the environment and is also sample efﬁcient to train.

Original: We provide two instantiations of our approach (ReBAL and GrBAL), and we provide a comparison with other prior methods on a range of continuous control tasks.
Revised: We provide two instantiations of our approach (ReBAL and GrBAL) and we provide a comparison with other prior methods on a range of continuous control tasks.

Original: We show that our approach is practical for real-world in contrast to less efﬁcient model-free meta-reinforcement learning approaches, and that the capability to adapt quickly is particularly important under complex real-world dynamics.
Revised: We show that our approach is practical for real-world in contrast to less efﬁcient model-free meta-reinforcement learning approaches and that the capability to adapt quickly is particularly important under complex real-world dynamics

Original: A E NVIRONMENTS
Revised: A E NVIRONMENTS

Original: Here we present all the detailed tasks an environments that are used in the result sections in the
Revised: Here we present all the detailed tasks an environments that are used in the result sections in the

Original: Appendix.
Revised: Appendix.

Original: Half-cheetah (HC): disabled joint.
Revised: Half-cheetah (HC): disabled joint.

Original: For each meta-training rollout, we randomly sample a joint to be disabled (i.e., the agent cannot apply torques to that joint).
Revised: For each meta-training rollout, we randomly sample a joint to be disabled (i.e., the agent cannot apply torques to that joint).

Original: At test time, we evaluate performance in three different situations: ( a ) disabling a joint seen at train time, ( b ) disabling an unseen joint, and ( c ) switching between disabled joints during a rollout.
Revised: At test time, we evaluate performance in three different situations: ( a ) disabling a joint seen at train time, ( b ) disabling an unseen joint, and ( c ) switching between disabled joints during a rollout.

Original: HC: sloped terrain.
Revised: HC: sloped terrain.

Original: During meta-training, we choose terrain of varying gentle upward and downward slopes.
Revised: During meta-training, we choose terrain of varying gentle upward and downward slopes.

Original: In this task, it is especially important to incorporate past experience into the model, since the cheetah has no means of directly observing the incline.
Revised: In this task, it is especially important to incorporate past experience into the model, since the cheetah has no means of directly observing the incline.

Original: At test time, we evaluate performance on ( d ) a gentle upward slope, ( e ) a steep hill that goes up and down, and ( f ) a steep upward slope.
Revised: At test time, we evaluate performance on ( d ) a gentle upward slope, ( e ) a steep hill that goes up and down, and ( f ) a steep upward slope.

Original: HC: pier.
Revised: HC: pier.

Original: In this task, the cheetah runs over a series of blocks that are ﬂoating on water.
Revised: In this task, the cheetah runs over a series of blocks that are ﬂoating on water.

Original: Each block moves up and down when stepped on, and the changes in the dynamics are drastic and rapid, due to each block having different damping and friction properties.
Revised: Each block moves up and down when stepped on, and the changes in the dynamics are drastic and rapid, due to each block having different damping and friction properties.

Original: The HC is meta-trained on varying these block properties, and tested on ( g ) a speciﬁc conﬁguration of block properties.
Revised: The HC is meta-trained on varying these block properties, and tested on ( g ) a speciﬁc conﬁguration of block properties.

Original: Ant: crippled leg.
Revised: Ant: crippled leg.

Original: For each meta-training rollout, we randomly sample a leg on a quadrupedal robot and disable it.
Revised: For each meta-training rollout, we randomly sample a leg on a quadrupedal robot and disable it.

Original: Disabling a leg unexpected drastically changes the dynamics.
Revised: Disabling a leg unexpected drastically changes the dynamics.

Original: We evaluate on ( h ) crippling a leg from the training distribution, ( i ) crippling a leg from outside the training distribution, and ( j ) crippling a leg in the middle of a rollout.
Revised: We evaluate on ( h ) crippling a leg from the training distribution, ( i ) crippling a leg from outside the training distribution, and ( j ) crippling a leg in the middle of a rollout.

Original: 7-DoF arm: force perturbations.
Revised: 7-DoF arm: force perturbations.

Original: We train a 7-DoF robot arm to carry an object to a goal position while applying random perturbation forces to the object.
Revised: We train a 7-DoF robot arm to carry an object to a goal position while applying random perturbation forces to the object.

Original: At test time, we evaluate with ( k ) a constant low force to the object, ( l ) a force 3 × stronger than during training, and ( m ) a force that randomly changes every 50 time-steps.
Revised: At test time, we evaluate with ( k ) a constant low force to the object, ( l ) a force 3 × stronger than during training, and ( m ) a force that randomly changes every 50 time-steps.

Original: This allow us to evaluate the ability of our method to adapt online to perturbations that clearly lie outside the training distribution.
Revised: This allow us to evaluate the ability of our method to adapt online to perturbations that clearly lie outside the training distribution.

Original: B M ODEL P REDICTION E RRORS : P RE - UPDATE VS . P OST - UPDATE
Revised: B M ODEL P REDICTION E RRORS : P RE - UPDATE VS . P OST - UPDATE

Original: In this section, we show the effect of adaptation in the case of GrBAL.
Revised: In this section, we show the effect of adaptation in the case of GrBAL.

Original: In particular, we show the histogram of the K step normalized error, as well as the per-timestep visualization of this error during a trajectory.
Revised: In particular, we show the histogram of the K step normalized error, as well as the per-timestep visualization of this error during a trajectory.

Original: Across all tasks and environments, the post-updated model ˆ p θ (cid:48)∗ achieves lower prediction error than the pre-updted model ˆ p θ ∗ .
Revised: Across all tasks and environments, the post-updated model ˆ p θ (cid:48)∗ achieves lower prediction error than the pre-updted model ˆ p θ ∗ .

Original: C E FFECT OF M ETA -T RAINING D ISTRIBUTION
Revised: C E FFECT OF M ETA -T RAINING D ISTRIBUTION

Original: To see how training distribution affects test performance, we ran an experiment that used GrBAL to train models of the 7-DOF arm, where each model was trained on the same number of datapoints during meta-training, but those datapoints came from different ranges of force perturbations.
Revised: To see how training distribution affects test performance, we ran an experiment that used GrBAL to train models of the 7-DOF arm, where each model was trained on the same number of datapoints during meta-training, but those datapoints came from different ranges of force perturbations.

Original: We observe (in the plot below)
Revised: We observe (in the plot below)

Original: that
Revised: that

Original: Seeing more during training is helpful during testing — a model that saw a large range of force perturbations during training performed the best 2.
Revised: Seeing more during training is helpful during testing — a model that saw a large range of force perturbations during training performed the best 2.

Original: A model that saw no perturbation forces during training did the worst 3.
Revised: A model that saw no perturbation forces during training did the worst 3.

Original: The middle 3 models show comparable performance in the "constant force = 4" case, which is an out-of-distribution task for those models.
Revised: The middle 3 models show comparable performance in the "constant force = 4" case, which is an out-of-distribution task for those models.

Original: Thus, there is not actually a strong restriction on what needs to be seen during training in order for adaptation to occur at train time (though there is a general trend that more is better)
Revised: Thus, there is not actually a strong restriction on what needs to be seen during training in order for adaptation to occur at train time (though there is a general trend that more is better)

Original: D R EWARD FUNCTIONS
Revised: D R EWARD FUNCTIONS

Original: For each MuJoCo agent, the same reward function is used across its various tasks.
Revised: For each MuJoCo agent, the same reward function is used across its various tasks.

Original: Table 2 shows the reward functions used for each agent.
Revised: Table 2 shows the reward functions used for each agent.

Original: We denote by x t the x-coordinate of the agent at time t , ee t refers to the position of the end-effector of the 7-DoF arm, and g corresponds to the position of the desired goal.
Revised: We denote by x t the x-coordinate of the agent at time t , ee t refers to the position of the end-effector of the 7-DoF arm, and g corresponds to the position of the desired goal.

Original: [Table]
Revised: [Table]

Original: E H YPERPARAMETERS
Revised: E H YPERPARAMETERS

Original: Below
Revised: Below

Original: [Table]
Revised: [Table]

Original: we list the hyperparameters of our experiments.
Revised: we list the hyperparameters of our experiments.

Original: In all experiments we used a single gradient step for the update rule of GrBAL.
Revised: In all experiments we used a single gradient step for the update rule of GrBAL.

Original: The learning rate (LR) of TRPO corresponds to the
Revised: The learning rate (LR) of TRPO corresponds to the

Original: Kullback–Leibler divergence constraint.
Revised: Kullback–Leibler divergence constraint.

Original: # Task/itr corresponds to the number of tasks sampled for collecting data to train the model or model, whereas # TS/itr is the total number of times steps collected (for all tasks).
Revised: # Task/itr corresponds to the number of tasks sampled for collecting data to train the model or model, whereas # TS/itr is the total number of times steps collected (for all tasks).

Original: Finally, T refers to the horizon of the task
Revised: Finally, T refers to the horizon of the task

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]