Original: P RO MP: P ROXIMAL M ETA -P OLICY S EARCH
Revised: P RO MP: P ROXIMAL M ETA -P OLICY S EARCH

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Credit assignment in Meta-reinforcement learning (Meta-RL) is still poorly understood.
Revised: Credit assignment in Meta-reinforcement learning (Meta-RL) is still poorly understood.

Original: Existing methods either neglect credit assignment to pre-adaptation behavior or implement it naively.
Revised: Existing methods either neglect credit assignment to pre-adaptation behavior or implement it naively.

Original: This leads to poor sample-efﬁciency during metatraining as well as ineffective task identiﬁcation strategies.
Revised: This leads to poor sample-efﬁciency during metatraining as well as ineffective task identiﬁcation strategies.

Original: This paper provides a theoretical analysis of credit assignment in gradient-based Meta-RL.
Revised: This paper provides a theoretical analysis of credit assignment in gradient-based Meta-RL.

Original: Building on the gained insights we develop a novel meta-learning algorithm that overcomes both the issue of poor credit assignment and previous difﬁculties in estimating meta-policy gradients.
Revised: Building on the gained insights we develop a novel meta-learning algorithm that overcomes both the issue of poor credit assignment and previous difﬁculties in estimating meta-policy gradients.

Original: By controlling the statistical distance of both pre-adaptation and adapted policies during meta-policy search, the proposed algorithm endows efﬁcient and stable meta-learning.
Revised: By controlling the statistical distance of both pre-adaptation and adapted policies during meta-policy search, the proposed algorithm endows efﬁcient and stable meta-learning.

Original: Our approach leads to superior pre-adaptation policy behavior and consistently outperforms previous Meta-RL algorithms in sample-efﬁciency, wall-clock time, and asymptotic performance.
Revised: Our approach leads to superior pre-adaptation policy behavior and consistently outperforms previous Meta-RL algorithms in sample-efﬁciency, wall-clock time, and asymptotic performance.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: A remarkable trait of human intelligence is the ability to adapt to new situations in the face of limited experience.
Revised: A remarkable trait of human intelligence is the ability to adapt to new situations in the face of limited experience.

Original: In contrast, our most successful artiﬁcial agents struggle in such scenarios.
Revised: In contrast, our most successful artiﬁcial agents struggle in such scenarios.

Original: While achieving impressive results, they suffer from high sample complexity in learning even a single task, fail to generalize to new situations, and require large amounts of additional data to successfully adapt to new environments.
Revised: While achieving impressive results, they suffer from high sample complexity in learning even a single task, fail to generalize to new situations, and require large amounts of additional data to successfully adapt to new environments.

Original: Meta-learning addresses these shortcomings by learning how to learn.
Revised: Meta-learning addresses these shortcomings by learning how to learn.

Original: Its objective is to learn an algorithm that allows the artiﬁcial agent to succeed in an unseen task when only limited experience is available, aiming to achieve the same fast adaptation that humans possess (Schmidhuber, 1987; Thrun & Pratt, 1998).
Revised: Its objective is to learn an algorithm that allows the artiﬁcial agent to succeed in an unseen task when only limited experience is available, aiming to achieve the same fast adaptation that humans possess (Schmidhuber, 1987; Thrun & Pratt, 1998).

Original: Despite recent progress, deep reinforcement learning (RL) still relies heavily on hand-crafted features and reward functions as well as engineered problem speciﬁc inductive bias.
Revised: Despite recent progress, deep reinforcement learning (RL) still relies heavily on hand-crafted features and reward functions as well as engineered problem speciﬁc inductive bias.

Original: Meta-RL aims to forego such reliance by acquiring inductive bias in a data-driven manner.
Revised: Meta-RL aims to forego such reliance by acquiring inductive bias in a data-driven manner.

Original: Recent work proves this approach to be promising, demonstrating that Meta-RL allows agents to obtain a diverse set of skills, attain better exploration strategies, and learn faster through meta-learned dynamics models or synthetic returns (Duan et al., 2016; Xu et al., 2018; Gupta et al., 2018b; Saemundsson et al., 2018).
Revised: Recent work proves this approach to be promising, demonstrating that Meta-RL allows agents to obtain a diverse set of skills, attain better exploration strategies, and learn faster through meta-learned dynamics models or synthetic returns (Duan et al., 2016; Xu et al., 2018; Gupta et al., 2018b; Saemundsson et al., 2018).

Original: Meta-RL is a multi-stage process in which the agent, after a few sampled environment interactions, adapts its behavior to the given task.
Revised: Meta-RL is a multi-stage process in which the agent, after a few sampled environment interactions, adapts its behavior to the given task.

Original: Despite its wide utilization, little work has been done to promote theoretical understanding of this process, leaving Meta-RL grounded on unstable foundations.
Revised: Despite its wide utilization, little work has been done to promote theoretical understanding of this process, leaving Meta-RL grounded on unstable foundations.

Original: Although the behavior prior to the adaptation step is instrumental for task identiﬁcation, the interplay between pre-adaptation sampling and posterior performance of the policy remains poorly understood.
Revised: Although the behavior prior to the adaptation step is instrumental for task identiﬁcation, the interplay between pre-adaptation sampling and posterior performance of the policy remains poorly understood.

Original: In fact, prior work in gradient-based Meta-RL has either entirely neglected credit assignment to the pre-update distribution (Finn et al., 2017) or implemented such credit assignment in a naive way (Al-Shedivat et al., 2018; Stadie et al., 2018).
Revised: In fact, prior work in gradient-based Meta-RL has either entirely neglected credit assignment to the pre-update distribution (Finn et al., 2017) or implemented such credit assignment in a naive way (Al-Shedivat et al., 2018; Stadie et al., 2018).

Original: To our knowledge, we provide the ﬁrst formal in-depth analysis of credit assignment w.r.t.
Revised: To our knowledge, we provide the ﬁrst formal in-depth analysis of credit assignment w.r.t.

Original: preadaptation sampling distribution in Meta-RL.
Revised: preadaptation sampling distribution in Meta-RL.

Original: Based on our ﬁndings, we develop a novel Meta-RL algorithm.
Revised: Based on our ﬁndings, we develop a novel Meta-RL algorithm.

Original: First, we analyze two distinct methods for assigning credit to pre-adaptation behavior.
Revised: First, we analyze two distinct methods for assigning credit to pre-adaptation behavior.

Original: We show that the recent formulation introduced by Al-Shedivat et al.
Revised: We show that the recent formulation introduced by Al-Shedivat et al.

Original: (2018) and Stadie et al.
Revised: (2018) and Stadie et al.

Original: (2018) leads to poor credit assignment, while the MAML formulation (Finn et al., 2017) potentially yields superior meta-policy updates.
Revised: (2018) leads to poor credit assignment, while the MAML formulation (Finn et al., 2017) potentially yields superior meta-policy updates.

Original: Second, based on insights from our formal analysis, we highlight both the importance and difﬁculty of proper meta-policy gradient estimates.
Revised: Second, based on insights from our formal analysis, we highlight both the importance and difﬁculty of proper meta-policy gradient estimates.

Original: In light of this, we propose the low variance curvature (LVC) surrogate objective which yields gradient estimates with a favorable bias-variance trade-off.
Revised: In light of this, we propose the low variance curvature (LVC) surrogate objective which yields gradient estimates with a favorable bias-variance trade-off.

Original: Finally, building upon the LVC estimator we develop Proximal MetaPolicy Search (ProMP), an efﬁcient and stable meta-learning algorithm for RL.
Revised: Finally, building upon the LVC estimator we develop Proximal MetaPolicy Search (ProMP), an efﬁcient and stable meta-learning algorithm for RL.

Original: In our experiments, we show that ProMP consistently outperforms previous Meta-RL algorithms in sample-efﬁciency, wall-clock time, and asymptotic performance.
Revised: In our experiments, we show that ProMP consistently outperforms previous Meta-RL algorithms in sample-efﬁciency, wall-clock time, and asymptotic performance.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Meta-Learning concerns the question of “learning to learn”, aiming to acquire inductive bias in a data driven manner, so that the learning process in face of unseen data or new problem settings is accelerated (Schmidhuber, 1987; Schmidhuber et al., 1997; Thrun & Pratt, 1998).
Revised: Meta-Learning concerns the question of “learning to learn”, aiming to acquire inductive bias in a data driven manner, so that the learning process in face of unseen data or new problem settings is accelerated (Schmidhuber, 1987; Schmidhuber et al., 1997; Thrun & Pratt, 1998).

Original: This can be achieved in various ways.
Revised: This can be achieved in various ways.

Original: One category of methods attempts to learn the “learning program” of an universal Turing machine in form of a recurrent / memory-augmented model that ingests datasets and either outputs the parameters of the trained model (Hochreiter et al., 2001; Andrychowicz et al., 2016; Chen et al., 2017; Ravi & Larochelle, 2017) or directly outputs predictions for given test inputs (Duan et al., 2016; Santoro et al., 2016;
Revised: One category of methods attempts to learn the “learning program” of an universal Turing machine in form of a recurrent / memory-augmented model that ingests datasets and either outputs the parameters of the trained model (Hochreiter et al., 2001; Andrychowicz et al., 2016; Chen et al., 2017; Ravi & Larochelle, 2017) or directly outputs predictions for given test inputs (Duan et al., 2016; Santoro et al., 2016;

Original: Mishra et al., 2018).
Revised: Mishra et al., 2018).

Original: Though very ﬂexible and capable of learning very efﬁcient adaptations, such methods lack performance guarantees and are difﬁcult to train on long sequences that arise in Meta-RL.
Revised: Though very ﬂexible and capable of learning very efﬁcient adaptations, such methods lack performance guarantees and are difﬁcult to train on long sequences that arise in Meta-RL.

Original: Another set of methods embeds the structure of a classical learning algorithm in the meta-learning procedure, and optimizes the parameters of the embedded learner during the meta-training (H¨usken & Goerick, 2000; Finn et al., 2017; Nichol et al., 2018; Miconi et al., 2018).
Revised: Another set of methods embeds the structure of a classical learning algorithm in the meta-learning procedure, and optimizes the parameters of the embedded learner during the meta-training (H¨usken & Goerick, 2000; Finn et al., 2017; Nichol et al., 2018; Miconi et al., 2018).

Original: A particular instance of the latter that has proven to be particularly successful in the context of RL is gradient-based metalearning (Finn et al., 2017; Al-Shedivat et al., 2018; Stadie et al., 2018).
Revised: A particular instance of the latter that has proven to be particularly successful in the context of RL is gradient-based metalearning (Finn et al., 2017; Al-Shedivat et al., 2018; Stadie et al., 2018).

Original: Its objective is to learn an initialization such that after one or few steps of policy gradients the agent attains full performance on a new task.
Revised: Its objective is to learn an initialization such that after one or few steps of policy gradients the agent attains full performance on a new task.

Original: A desirable property of this approach is that even if fast adaptation fails, the agent just falls back on vanilla policy-gradients.
Revised: A desirable property of this approach is that even if fast adaptation fails, the agent just falls back on vanilla policy-gradients.

Original: However, as we show, previous gradient-based Meta-RL methods either neglect or perform poor credit assignment w.r.t.
Revised: However, as we show, previous gradient-based Meta-RL methods either neglect or perform poor credit assignment w.r.t.

Original: the pre-update sampling distribution.
Revised: the pre-update sampling distribution.

Original: A diverse set of methods building on meta-RL, has recently been introduced.
Revised: A diverse set of methods building on Meta-RL, has recently been introduced.

Original: This includes: learning exploration strategies (Gupta et al., 2018b), synthetic rewards (Sung et al., 2017; Xu et al., 2018), unsupervised policy acquisition (Gupta et al., 2018a), model-based RL (Clavera et al., 2018;
Revised: This includes: learning exploration strategies (Gupta et al., 2018b), synthetic rewards (Sung et al., 2017; Xu et al., 2018), unsupervised policy acquisition (Gupta et al., 2018a), model-based RL (Clavera et al., 2018;

Original: Saemundsson et al., 2018), learning in competitive environments (Al-Shedivat et al., 2018) and meta-learning modular policies (Frans et al., 2018;
Revised: Saemundsson et al., 2018), learning in competitive environments (Al-Shedivat et al., 2018) and meta-learning modular policies (Frans et al., 2018;

Original: Alet et al., 2018).
Revised: Alet et al., 2018).

Original: Many of the mentioned approaches build on previous gradient-based meta-learning methods that insufﬁciently account for the pre-update distribution.
Revised: Many of the mentioned approaches build on previous gradient-based meta-learning methods that insufﬁciently account for the pre-update distribution.

Original: ProMP overcomes these deﬁciencies, providing the necessary framework for novel applications of Meta-RL in unsolved problems.
Revised: ProMP overcomes these deﬁciencies, providing the necessary framework for novel applications of Meta-RL in unsolved problems.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Reinforcement Learning.
Revised: Reinforcement Learning.

Original: A discrete-time ﬁnite Markov decision process (MDP), T , is deﬁned by the tuple ( S , A , p, p 0 , r, H ) .
Revised: A discrete-time ﬁnite Markov decision process (MDP), T , is deﬁned by the tuple ( S , A , p, p 0 , r, H ) .

Original: Here, S is the set of states, A the action space, p ( s t +1 | s t , a t ) the transition distribution, p 0 represents the initial state distribution, r : S ×
Revised: Here, S is the set of states, A the action space, p ( s t +1 | s t , a t ) the transition distribution, p 0 represents the initial state distribution, r : S ×

Original: A → R is a reward function, and H the time horizon.
Revised: A → R is a reward function, and H the time horizon.

Original: We omit the discount factor γ in the following elaborations for notational brevity.
Revised: We omit the discount factor γ in the following elaborations for notational brevity.

Original: However, it is straightforward to include it by substituting the reward by r ( s t , a t ) :
Revised: However, it is straightforward to include it by substituting the reward by r ( s t , a t ) :

Original: = γ t r ( s t , a t ) .
Revised: = γ t r ( s t , a t ) .

Original: We deﬁne the return R ( τ ) as the sum of rewards along a trajectory τ :=
Revised: We deﬁne the return R ( τ ) as the sum of rewards along a trajectory τ :=

Original: ( s 0 , a 0 , ..., s H − 1 , a H − 1 , s H ) .
Revised: ( s 0 , a 0 , ..., s H − 1 , a H − 1 , s H ) .

Original: The goal of reinforcement learning is to ﬁnd a policy π ( a | s ) that maximizes the expected return E τ ∼ P T ( τ | π )
Revised: The goal of reinforcement learning is to ﬁnd a policy π ( a | s ) that maximizes the expected return E τ ∼ P T ( τ | π )

Original: [ R ( τ )] .
Revised: [ R ( τ )] .

Original: Meta-Reinforcement Learning goes one step further, aiming to learn a learning algorithm which is able to quickly learn the optimal policy for a task T drawn from a distribution of tasks ρ ( T ) .
Revised: Meta-Reinforcement Learning goes one step further, aiming to learn a learning algorithm which is able to quickly learn the optimal policy for a task T drawn from a distribution of tasks ρ ( T ) .

Original: Each task T corresponds to a different MDP.
Revised: Each task T corresponds to a different MDP.

Original: Typically, it is assumed that the distribution of tasks share the action and state space, but may differ in their reward function or their dynamics.
Revised: Typically, it is assumed that the distribution of tasks share the action and state space, but may differ in their reward function or their dynamics.

Original: Gradient-based meta-learning aims to solve this problem by learning the parameters θ of a policy π θ such that performing a single or few steps of vanilla policy gradient (VPG) with the given task leads to the optimal policy for that task.
Revised: Gradient-based meta-learning aims to solve this problem by learning the parameters θ of a policy π θ such that performing a single or few steps of vanilla policy gradient (VPG) with the given task leads to the optimal policy for that task.

Original: This meta-learning formulation, also known under the name of MAML, was ﬁrst introduced by Finn et al.
Revised: This meta-learning formulation, also known under the name of MAML, was ﬁrst introduced by Finn et al.

Original: We refer to it as formulation I which can be expressed as maximizing the objective
Revised: We refer to it as formulation I which can be expressed as maximizing the objective

Original: [Equation]
Revised: [Equation]

Original: In that U denotes the update function which depends on the task T , and performs one VPG step towards maximizing the performance of the policy in T .
Revised: In that U denotes the update function which depends on the task T , and performs one VPG step towards maximizing the performance of the policy in T .

Original: For national brevity and conciseness we assume a single policy gradient adaptation step.
Revised: For national brevity and conciseness we assume a single policy gradient adaptation step.

Original: Nonetheless, all presented concepts can easily be extended to multiple adaptation steps.
Revised: Nonetheless, all presented concepts can easily be extended to multiple adaptation steps.

Original: Later work proposes a slightly different notion of gradient-based meta-RL, also known as E-MAML, that attempts to circumvent issues with the meta-gradient estimation in MAML (Al-Shedivat et al., 2018; Stadie et al., 2018):[Equation] N
Revised: Later work proposes a slightly different notion of gradient-based Meta-RL, also known as E-MAML, that attempts to circumvent issues with the meta-gradient estimation in MAML (Al-Shedivat et al., 2018; Stadie et al., 2018):

Original: [Equation]
Revised: [Equation]

Original:  n = (cid:104)[Equation] ( n
Revised: N (cid:88) n = (cid:104)

Original: [Equation]
Revised: [Equation]

Original: (cid:105)
Revised: (cid:105)

Original: Formulation II views U as a deterministic function that depends on N sampled trajectories from a speciﬁc task.
Revised: Formulation II views U as a deterministic function that depends on N sampled trajectories from a speciﬁc task.

Original: In contrast to formulation I, the expectation over pre-update trajectories τ is applied outside of the update function.
Revised: In contrast to formulation I, the expectation over pre-update trajectories τ is applied outside of the update function.

Original: Throughout this paper we refer to π θ as pre-update policy, and π θ (cid:48) as post-update policy.
Revised: Throughout this paper we refer to π θ as pre-update policy, and π θ (cid:48) as post-update policy.

Original:  S AMPLING D ISTRIBUTION C REDIT A SSIGNMENT
Revised:  S AMPLING D ISTRIBUTION C REDIT A SSIGNMENT

Original: This section analyzes the two gradient-based Meta-RL formulations introduced in Section 3.
Revised: This section analyzes the two gradient-based Meta-RL formulations introduced in Section 3.

Original: Figure 1 illustrates the stochastic computation graphs (Schulman et al., 2015b) of both formulations.
Revised: Figure 1 illustrates the stochastic computation graphs (Schulman et al., 2015b) of both formulations.

Original: The red arrows depict how credit assignment w.r.t the pre-update sampling distribution P T ( τ | θ ) is propagated.
Revised: The red arrows depict how credit assignment w.r.t the pre-update sampling distribution P T ( τ | θ ) is propagated.

Original: Formulation I (left) propagates the credit assignment through the update step, thereby exploiting the full problem structure.
Revised: Formulation I (left) propagates the credit assignment through the update step, thereby exploiting the full problem structure.

Original: In contrast, formulation II (right) neglects the inherent structure, directly assigning credit from post-update return R (cid:48) to the pre-update policy π θ which leads to noisier, less effective credit assignment.
Revised: In contrast, formulation II (right) neglects the inherent structure, directly assigning credit from post-update return R (cid:48) to the pre-update policy π θ which leads to noisier, less effective credit assignment.

Original: Both formulations optimize for the same objective, and are equivalent at the 0 th order.
Revised: Both formulations optimize for the same objective, and are equivalent at the 0 th order.

Original: However, because of the difference in their formulation and stochastic computation graph, their gradients and the resulting optimization step differs.
Revised: However, because of the difference in their formulation and stochastic computation graph, their gradients and the resulting optimization step differs.

Original: In the following
Revised: In the following

Original: [Equation]
Revised: [Equation]

Original: we shed light on how and where formulation II loses signal by analyzing the gradients of both formulations, which can be written as (see
Revised: we shed light on how and where formulation II loses signal by analyzing the gradients of both formulations, which can be written as (see

Original: Appendix A for more details and derivations)
Revised: Appendix A for more details and derivations)

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term ∇ θ J post ( τ , τ (cid:48) ) is equal in both formulations
Revised: The ﬁrst term ∇ θ J post ( τ , τ (cid:48) ) is equal in both formulations

Original: [Equation]
Revised: [Equation]

Original: but the second term, ∇ θ J pre ( τ , τ (cid:48) ) , differs between them.
Revised: but the second term, ∇ θ J pre ( τ , τ (cid:48) ) , differs between them.

Original: In particular, they correspond to
Revised: In particular, they correspond to

Original: [Equation]
Revised: [Equation]

Original: ∇ θ J post ( τ , τ (cid:48) )
Revised: ∇ θ J post ( τ , τ (cid:48) )

Original: simply corresponds to a policy gradient step on the post-update policy π θ (cid:48) w.r.t θ (cid:48) , followed by a linear transformation from post- to pre-update parameters.
Revised: simply corresponds to a policy gradient step on the post-update policy π θ (cid:48) w.r.t θ (cid:48) , followed by a linear transformation from post- to pre-update parameters.

Original: It corresponds to increasing the likelihood of the trajectories τ (cid:48) that led to higher returns.
Revised: It corresponds to increasing the likelihood of the trajectories τ (cid:48) that led to higher returns.

Original: However, this term does not optimize for the pre-update sampling distribution, i.e., which trajectories τ led to better adaptation steps.
Revised: However, this term does not optimize for the pre-update sampling distribution, i.e., which trajectories τ led to better adaptation steps.

Original: The credit assignment w.r.t.
Revised: The credit assignment w.r.t.

Original: the pre-updated sampling distribution is carried out by the second term.
Revised: the pre-updated sampling distribution is carried out by the second term.

Original: In formulation II, ∇ θ J II pre can be viewed as standard reinforcement learning on π θ with R ( τ (cid:48) )
Revised: In formulation II, ∇ θ J II pre can be viewed as standard reinforcement learning on π θ with R ( τ (cid:48) )

Original: as reward signal, treating the update function U as part of the unknown dynamics of the system.
Revised: as reward signal, treating the update function U as part of the unknown dynamics of the system.

Original: This shifts the pre-update sampling distribution to better adaptation steps.
Revised: This shifts the pre-update sampling distribution to better adaptation steps.

Original: Formulation I takes the causal dependence of P T ( τ (cid:48) | θ (cid:48) ) on P T ( τ | θ ) into account.
Revised: Formulation I takes the causal dependence of P T ( τ (cid:48) | θ (cid:48) ) on P T ( τ | θ ) into account.

Original: It does so by maximizing the inner product of pre-update and post-update policy gradients (see Eq. 4).
Revised: It does so by maximizing the inner product of pre-update and post-update policy gradients (see Eq. 4).

Original: This steers the pre-update policy towards 1) larger post-updates returns 2) larger adaptation steps α ∇ θ J inner , 3) better alignment of pre- and post-update policy gradients (Li et al., 2017; Nichol et al., 2018).
Revised: This steers the pre-update policy towards 1) larger post-updates returns 2) larger adaptation steps α ∇ θ J inner , 3) better alignment of pre- and post-update policy gradients (Li et al., 2017; Nichol et al., 2018).

Original: When combined, these effects directly optimize for adaptation.
Revised: When combined, these effects directly optimize for adaptation.

Original: As a result, we expect the ﬁrst meta-policy gradient formulation, J I , to yield superior learning properties.
Revised: As a result, we expect the ﬁrst meta-policy gradient formulation, J I , to yield superior learning properties.

Original:  L OW V ARIANCE C URVATURE E STIMATOR
Revised:  L OW V ARIANCE C URVATURE E STIMATOR

Original: In the previous section we show that the formulation introduced by Finn et al.
Revised: In the previous section we show that the formulation introduced by Finn et al.

Original: (2017) results in superior meta-gradient updates, which should in principle lead to improved convergence properties.
Revised: (2017) results in superior meta-gradient updates, which should in principle lead to improved convergence properties.

Original: However, obtaining correct and low variance estimates of the respective meta-gradients proves challenging.
Revised: However, obtaining correct and low variance estimates of the respective meta-gradients proves challenging.

Original: As discussed by Foerster et al.
Revised: As discussed by Foerster et al.

Original: (2018), and shown in Appendix B.3, the score function surrogate objective approach is ill suited for calculating higher order derivatives via automatic differentiation toolboxes.
Revised: (2018), and shown in Appendix B.3, the score function surrogate objective approach is ill suited for calculating higher order derivatives via automatic differentiation toolboxes.

Original: This important fact was overlooked in the original RL-MAML implementation (Finn et al., 2017) leading to incorrect meta-gradient estimates 1 .
Revised: This important fact was overlooked in the original RL-MAML implementation (Finn et al., 2017) leading to incorrect meta-gradient estimates 1 .

Revised: As a result, ∇ θ J pre does not appear in the gradients of the meta-objective (i.e. ∇ θ J = ∇ θ J post ).

Revised: Hence, MAML does not perform any credit assignment to pre-adaptation behavior.

Original: But, even when properly implemented, we show that these gradients exhibit high variance.
Revised: But, even when properly implemented, we show that the meta-gradients exhibit high variance.

Original: Speciﬁcally, the estimation of the hessian of the RL-objective, which is inherent in the metagradients, requires special consideration.
Revised: Speciﬁcally, the estimation of the hessian of the RL-objective, which is inherent in the metagradients, requires special consideration.

Original: In this section, we motivate and introduce the low variance curvature estimator (LVC): an improved estimator for the hessian of the RL-objective which promotes better meta-policy gradient updates.
Revised: In this section, we motivate and introduce the low variance curvature estimator (LVC): an improved estimator for the hessian of the RL-objective which promotes better meta-policy gradient updates.

Original: As we show in Appendix A.1, we can write the gradient of the meta-learning objective as
Revised: As we show in Appendix A.1, we can write the gradient of the meta-learning objective as

Original: [Equation]
Revised: [Equation]

Original: Since the update function U resembles a policy gradient step, its gradient ∇ θ U ( θ, T ) involves computing the hessian of the reinforcement learning objective, i [Equation] e., ∇ 2 θ E τ ∼ P T ( τ | θ )
Revised: Since the update function U resembles a policy gradient step, its gradient ∇ θ U ( θ, T ) involves computing the hessian of the reinforcement learning objective, i.e., ∇ 2 θ E τ ∼ P T ( τ | θ )

Original: [ R ( τ )] .
Revised: [ R ( τ )] .

Original: Estimating this hessian has been discussed in Baxter & Bartlett (2001) and Furmston et al.
Revised: Estimating this hessian has been discussed in Baxter & Bartlett (2001) and Furmston et al.

Original: In the inﬁnite horizon MDP case, Baxter & Bartlett (2001) derived a decomposition of the hessian.
Revised: In the inﬁnite horizon MDP case, Baxter & Bartlett (2001) derived a decomposition of the hessian.

Original: We extend their ﬁnding to the ﬁnite horizon case, showing that the hessian can be decomposed into three matrix terms (see Appendix B.2 for proof):
Revised: We extend their ﬁnding to the ﬁnite horizon case, showing that the hessian can be decomposed into three matrix terms (see Appendix B.2 for proof):

Original: [Equation]
Revised: [Equation]

Original: 6) whereby
Revised: whereby

Original: [Equation]
Revised: [Equation]

Revised: t = ∇ 2 θ log π θ ( a t , s t ) (cid:32)

Revised: H − 1 (cid:88) t (cid:48)

Revised: =

Revised: t r ( s t (cid:48) , a t (cid:48) ) (cid:33)(cid:35)

Revised: H 12 =

Revised: E τ ∼ P T ( τ | θ ) (cid:34) H − 1 (cid:88) t = ∇ θ log π θ ( a t , s t ) ∇ θ Q π θ t ( s t , a t ) (cid:62) (cid:35)

Original: Here Q π θ t ( s t , a t ) =
Revised: Here Q π θ t ( s t , a t ) =

Original: E τ t +1: H − 1 ∼ P T ( ·| θ ) (cid:104)(cid:80)
Revised: E τ t +1: H − 1 ∼ P T ( ·| θ ) (cid:104)(cid:80)

Original: H − 1 t (cid:48)
Revised: H − 1 t (cid:48)

Original: =
Revised: =

Original: t r ( s t (cid:48) , a t (cid:48) ) | s t , a t (cid:105) denotes the expected state-action value function under policy π θ at time t .
Revised: t r ( s t (cid:48) , a t (cid:48) ) | s t , a t (cid:105) denotes the expected state-action value function under policy π θ at time t .

Original: Computing the expectation of the RL-objective is in general intractable.
Revised: Computing the expectation of the RL-objective is in general intractable.

Original: Typically, its gradients are computed with a Monte Carlo estimate based on the policy gradient theorem (Eq. 82).
Revised: Typically, its gradients are computed with a Monte Carlo estimate based on the policy gradient theorem (Eq. 82).

Original: In practical implementations, such an estimate is obtained by automatically differentiating a surrogate objective (Schulman et al., 2015b).
Revised: In practical implementations, such an estimate is obtained by automatically differentiating a surrogate objective (Schulman et al., 2015b).

Original: However, this results in a highly biased hessian estimate which just computes H 2 , entirely dropping the terms H 1 and H 

Revised: However, this results in a highly biased hessian estimate which just computes H 2 , entirely dropping the terms H 1 and H

Original: + H (cid:62) 12 .
Revised: + H (cid:62) 12 .

Original: In the notation of the previous section, it leads to neglecting the ∇ θ J pre term, ignoring the inﬂuence of the pre-update sampling distribution.
Revised: In the notation of the previous section, it leads to neglecting the ∇ θ J pre term, ignoring the inﬂuence of the pre-update sampling distribution.

Original: The issue can be overcome using the DiCE formulation, which allows to compute unbiased higherorder Monte Carlos estimates of arbitrary stochastic computation graphs (Foerster et al., 2018).
Revised: The issue can be overcome using the DiCE formulation, which allows to compute unbiased higherorder Monte Carlos estimates of arbitrary stochastic computation graphs (Foerster et al., 2018).

Original: The
Revised: The

Original: DiCE-RL objective can be rewritten as follows
Revised: DiCE-RL objective can be rewritten as follows

Original: [Equation]
Revised: [Equation]

Original: In that, ⊥ denotes the “stop gradient” operator, i.e., ⊥ ( f θ ( x ))
Revised: In that, ⊥ denotes the “stop gradient” operator, i.e., ⊥ ( f θ ( x ))

Original: → f θ ( x ) but ∇ θ ⊥ ( f θ ( x ))
Revised: → f θ ( x ) but ∇ θ ⊥ ( f θ ( x ))

Original: → 0 .
Revised: → 0 .

Original: The sequential dependence of π θ ( a t | s t ) within the trajectory, manifesting itself through the product of importance weights in (7), results in high variance estimates of the hessian ∇ 2 θ E τ ∼ P T [Equation] τ | θ )
Revised: The sequential dependence of π θ ( a t | s t ) within the trajectory, manifesting itself through the product of importance weights in (7), results in high variance estimates of the hessian ∇ 2 θ E τ ∼ P T ( τ | θ )

Original: [ R ( τ )] .
Revised: [ R ( τ )] .

Original: As noted by Furmston et al. (2016), H 12 is particularly difﬁcult to estimate, since it involves three nested sums along the trajectory.
Revised: As noted by Furmston et al. (2016), H 12 is particularly difﬁcult to estimate, since it involves three nested sums along the trajectory.

Original: In section 7.2 we empirically show that the high variance estimates of the DiCE objective lead to noisy meta-policy gradients and poor learning performance.
Revised: In section 7.2 we empirically show that the high variance estimates of the DiCE objective lead to noisy meta-policy gradients and poor learning performance.

Original: To facilitate a sample efﬁcient meta-learning, we introduce the low variance curvature (LVC) estimator:
Revised: To facilitate a sample efﬁcient meta-learning, we introduce the low variance curvature (LVC) estimator:

Original: [Equation]
Revised: [Equation]

Original: By removing the sequential dependence of π θ ( a t
Revised: By removing the sequential dependence of π θ ( a t

Original: | s t ) within trajectories, the hessian estimate neglects the term H 

Revised: | s t ) within trajectories, the hessian estimate neglects the term H

Original: + H (cid:62) 12 which leads to a variance reduction, but makes the estimate biased.
Revised: + H (cid:62) 12 which leads to a variance reduction, but makes the estimate biased.

Original: The choice of this objective function is motivated by ﬁndings in Furmston et al.
Revised: The choice of this objective function is motivated by ﬁndings in Furmston et al.

Original: (2016): under certain conditions the term H 

Revised: (2016): under certain conditions the term H

Original: + H (cid:62) 12 vanishes around local optima θ ∗ , i.e., E τ [ ∇ 2 θ J LVC ]
Revised: + H (cid:62) 12 vanishes around local optima θ ∗ , i.e., E τ [ ∇ 2 θ J LVC ]

Original: → E τ [ ∇ 2 θ J DiCE ] as θ → θ ∗ .
Revised: → E τ [ ∇ 2 θ J DiCE ] as θ → θ ∗ .

Original: Hence, the bias of the LVC estimator becomes negligible close to local optima.
Revised: Hence, the bias of the LVC estimator becomes negligible close to local optima.

Original: The experiments in section 7.2 underpin the theoretical ﬁndings, showing that the low variance hessian estimates obtained through J LVC improve the sample-efﬁciency of meta-learning by a signiﬁcant margin when compared to J DiCE .
Revised: The experiments in section 7.2 underpin the theoretical ﬁndings, showing that the low variance hessian estimates obtained through J LVC improve the sample-efﬁciency of meta-learning by a signiﬁcant margin when compared to J DiCE .

Original: We refer the interested reader to Appendix B for derivations and a more detailed discussion.
Revised: We refer the interested reader to Appendix B for derivations and a more detailed discussion.

Original:  P RO MP: P ROXIMAL M ETA -P OLICY S EARCH
Revised:  P RO MP: P ROXIMAL M ETA -P OLICY S EARCH

Original: Building on the previous sections, we develop a novel meta-policy search method based on the low variance curvature objective which aims to solve the following optimization problem:
Revised: Building on the previous sections, we develop a novel meta-policy search method based on the low variance curvature objective which aims to solve the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: with θ (cid:48)

Original: [Equation]

Original: =

Original: [Equation]

Original: + [Equation] [Equation] Prior work has optimized this objective using either vanilla policy gradient (VPG) or TRPO (Schulman et al., 2015a).
Revised: Prior work has optimized this objective using either vanilla policy gradient (VPG) or TRPO (Schulman et al., 2015a).

Original: TRPO holds the promise to be more data efﬁcient and stable during the learning process when compared to VPG.
Revised: TRPO holds the promise to be more data efﬁcient and stable during the learning process when compared to VPG.

Original: However, it requires computing the Fisher information matrix (FIM).
Revised: However, it requires computing the Fisher information matrix (FIM).

Original: Estimating the FIM is particularly problematic in the meta-learning set up.
Revised: Estimating the FIM is particularly problematic in the meta-learning set up.

Original: The meta-policy gradients already involve second order derivatives; as a result, the time complexity of the FIM estimate is cubic in the number of policy parameters.
Revised: The meta-policy gradients already involve second order derivatives; as a result, the time complexity of the FIM estimate is cubic in the number of policy parameters.

Original: Typically, the problem is circumvented using ﬁnite difference methods, which introduce further approximation errors.
Revised: Typically, the problem is circumvented using ﬁnite difference methods, which introduce further approximation errors.

Original: The recently introduced PPO algorithm (Schulman et al., 2017) achieves comparable results to
Revised: The recently introduced PPO algorithm (Schulman et al., 2017) achieves comparable results to

Original: TRPO with the advantage of being a ﬁrst order method.
Revised: TRPO with the advantage of being a ﬁrst order method.

Original: PPO uses a surrogate clipping objective which allows it to safely take multiple gradient steps without re-sampling trajectories. 
Revised: PPO uses a surrogate clipping objective which allows it to safely take multiple gradient steps without re-sampling trajectories. [Table]

Original: [Equation]
Revised: [Equation]

Original: In case of Meta-RL, it does not sufﬁce to just replace the post-update reward objective with J CLIP T .
Revised: In case of Meta-RL, it does not sufﬁce to just replace the post-update reward objective with J CLIP T .

Original: In order to safely perform multiple meta-gradient steps based on the same sampled data from a recent policy π θ o , we also need to 1) account for changes in the pre-update action distribution π θ ( a t | s t ) , and 2) bound changes in the pre-update state visitation distribution (Kakade & Langford, 2002).
Revised: In order to safely perform multiple meta-gradient steps based on the same sampled data from a recent policy π θ o , we also need to 1) account for changes in the pre-update action distribution π θ ( a t | s t ) , and 2) bound changes in the pre-update state visitation distribution (Kakade & Langford, 2002).

Original: We propose Proximal Meta-Policy Search (ProMP) which incorporates both the beneﬁts of proximal policy optimization and the low variance curvature objective (see Alg. 1.)

Original: In order to comply with

Original: Require: Task distribution ρ , step sizes α , β , KL-penalty coefﬁcient η , clipping range (cid:15) 1: Randomly initialize θ 2: while θ not converged do 3:
Revised: Require: Task distribution ρ , step sizes α , β , KL-penalty coefﬁcient η , clipping range (cid:15) 1: Randomly initialize θ 2: while θ not converged do 3:

Original: Sample batch of tasks T i ∼ ρ ( T ) 4: for step n = 0 , ..., N − 1 do 5: if n = 0 then 6:
Revised: Sample batch of tasks T i ∼ ρ ( T ) 4: for step n = 0 , ..., N − 1 do 5: if n = 0 then 6:

Original: Set θ o ← θ 7: for all T i ∼ ρ ( T ) do 8:
Revised: Set θ o ← θ 7: for all T i ∼ ρ ( T ) do 8:

Original: Sample pre-update trajectories D i
Revised: Sample pre-update trajectories D i

Original: = { τ i } from T i using π θ 9:
Revised: = { τ i } from T i using π θ 9:

Original: Compute adapted parameters θ (cid:48)
Revised: Compute adapted parameters θ (cid:48)

Original: o,i ← θ + α ∇ θ J LR T i ( θ ) with D i = { τ i } 10:
Revised: o,i ← θ + α ∇ θ J LR T i ( θ ) with D i = { τ i } 10:

Original: Sample post-update trajectories D (cid:48)
Revised: Sample post-update trajectories D (cid:48)

Original: i = { τ (cid:48) i } from T i using π θ (cid:48)
Revised: i = { τ (cid:48) i } from T i using π θ (cid:48)

Original: o,i 11:
Revised: o,i 11:

Original: Update θ ← θ + β (cid:80) T i ∇ θ J ProMP T i ( θ ) using each D (cid:48) 
Revised: Update θ ← θ + β (cid:80) T i ∇ θ J ProMP T i ( θ ) using each D (cid:48) i = { τ (cid:48) i }

Revised: We propose Proximal Meta-Policy Search (ProMP) which incorporates both the beneﬁts of proximal policy optimization and the low variance curvature objective (see Alg. 1.)

Original: i = { τ (cid:48) i } requirement 1), ProMP replaces the “stop gradient” importance weight π θ ( a t | s t ) ⊥
Revised: In order to comply with requirement 1), ProMP replaces the “stop gradient” importance weight π θ ( a t | s t ) ⊥

Original: ( π θ ( a t | s t )) by the likelihood ratio π θ ( a t | s t )
Revised: ( π θ ( a t | s t )) by the likelihood ratio π θ ( a t | s t )

Original: π θo ( a t | s t )) , which results in the following objective
Revised: π θo ( a t | s t )) , which results in the following objective

Original: [Equation]
Revised: [Equation]

Original: (cid:35) (12) An important feature of this objective is that its derivatives  evaluated at θ o are identical to those of the LVC objective.
Revised: An important feature of this objective is that its derivatives w.r.t θ evaluated at θ o are identical to those of the LVC objective, and it additionally accounts for changes in the pre-update action distribution.

Original: To satisfy condition 2) we extend the clipped meta-objective with a KL-penalty term between π θ and π θ o .
Revised: To satisfy condition 2) we extend the clipped meta-objective with a KL-penalty term between π θ and π θ o .

Original: This KL-penalty term enforces a soft local “trust region” around π θ o , preventing the shift in state visitation distribution to become large during optimization.
Revised: This KL-penalty term enforces a soft local “trust region” around π θ o , preventing the shift in state visitation distribution to become large during optimization.

Original: This enables us to take multiple meta-policy gradient steps without re-sampling.
Revised: This enables us to take multiple meta-policy gradient steps without re-sampling.

Original: Altogether, ProMP optimizes
Revised: Altogether, ProMP optimizes

Original: [Equation]
Revised: [Equation]

Original: ProMP consolidates the insights developed throughout the course of this paper, while at the same time making maximal use of recently developed policy gradients algorithms.
Revised: ProMP consolidates the insights developed throughout the course of this paper, while at the same time making maximal use of recently developed policy gradients algorithms.

Original: First, its meta-learning formulation exploits the full structural knowledge of gradient-based meta-learning.
Revised: First, its meta-learning formulation exploits the full structural knowledge of gradient-based meta-learning.

Original: Second, it incorporates a low variance estimate of the RL-objective hessian.
Revised: Second, it incorporates a low variance estimate of the RL-objective hessian.

Original: Third, ProMP controls the statistical distance of both pre- and post-adaptation policies, promoting efﬁcient and stable meta-learning.
Revised: Third, ProMP controls the statistical distance of both pre- and post-adaptation policies, promoting efﬁcient and stable meta-learning.

Original: All in all, ProMP consistently outperforms previous gradient-based meta-RL algorithms in sample complexity, wall clock time, and asymptotic performance (see Section 7.1).
Revised: All in all, ProMP consistently outperforms previous gradient-based meta-RL algorithms in sample complexity, wall clock time, and asymptotic performance (see Section 7.1).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In order to empirically validate the theoretical arguments outlined above, this section provides a detailed experimental analysis that aims to answer the following questions: (i) How does ProMP perform against previous Meta-RL algorithms?
Revised: In order to empirically validate the theoretical arguments outlined above, this section provides a detailed experimental analysis that aims to answer the following questions: (i) How does ProMP perform against previous Meta-RL algorithms?

Original: (ii) How do the lower variance but biased LVC gradient estimates compare to the high variance, unbiased DiCE estimates?
Revised: (ii) How do the lower variance but biased LVC gradient estimates compare to the high variance, unbiased DiCE estimates?

Original: (iii) Do the different formulations result in different pre-update exploration properties?
Revised: (iii) Do the different formulations result in different pre-update exploration properties?

Original: (iv) How do formulation I and formulation II differ in their meta-gradient estimates and convergence properties?
Revised: (iv) How do formulation I and formulation II differ in their meta-gradient estimates and convergence properties?

Original: To answer the posed questions, we evaluate our approach on six continuous control Meta-RL benchmark environments based on OpenAI Gym and the Mujoco simulator (Brockman et al., 2016;
Revised: To answer the posed questions, we evaluate our approach on six continuous control Meta-RL benchmark environments based on OpenAI Gym and the Mujoco simulator (Brockman et al., 2016;

Original: Todorov et al., 2012).
Revised: Todorov et al., 2012).

Original: A description of the experimental setup is found in Appendix D. In all experiments, the reported curves are averaged over at least three random seeds.
Revised: A description of the experimental setup is found in Appendix D. In all experiments, the reported curves are averaged over at least three random seeds.

Original: Returns are estimated based on sampled trajectories from the adapted post-update policies and averaged over sampled tasks.
Revised: Returns are estimated based on sampled trajectories from the adapted post-update policies and averaged over sampled tasks.

Original: The source code and the experiments data are available on our supplementary website. 
Revised: The source code and the experiment data are available on our supplementary website. [Figure]

Original: 7. M ETA -G RADIENT B ASED C OMPARISON
Revised: 7. M ETA -G RADIENT B ASED C OMPARISON

Original: We compare our method, ProMP, in sample complexity and asymptotic performance to four other gradient-based approaches: TRPO-MAML (Finn et al., 2017), E-MAML-TRPO, E-MAMLVPG (Stadie et al., 2018), and LVC-VPG, an ablated version of our method that uses the LVC objective in the adaptation step and meta-optimizes with vanilla policy gradient.
Revised: We compare our method, ProMP, in sample complexity and asymptotic performance to the gradientbased meta-learning approaches MAML-TRPO (Finn et al., 2017) and E-MAML-TRPO (see Fig. Note that MAML corresponds to the original implementation of RL-MAML by (Finn et al., 2017) where no credit assignment to the pre-adaptation policy is happening (see Appendix B. for details).

Revised: Moreover, we provide a second study which focuses on the underlying meta-gradient estimator.

Revised: Speciﬁcally, we compare the LVC, DiCE, MAML and E-MAML estimators while optimizing meta-learning objective with vanilla policy gradient (VPG) ascent.

Original:  These algorithms are benchmarked on six different locomotion tasks that require adaptation: theant must learn to run in different directions in the 2D-plane, the half-cheetah and walker must switch between running forward and backward, the walker and hopper have to adapt to  different conﬁguration of their dynamics.
Revised: This can be viewed as an ablated version of the algorithms which tries to eliminate the inﬂuences of the outer optimizers on the learning performance (see Fig. 3) [Figure] These algorithms are benchmarked on six different locomotion tasks that require adaptation: the half-cheetah and walker must switch between running forward and backward, the high-dimensional agents ant and humanoid must learn to adapt to run in different directions in the 2D-plane [Figure]

Original: Finally, we present how these algorithms perform in a high-dimensional environment, where a humanoid has to adapt torun in different directions
Revised: and the hopper and walker have to adapt to different conﬁguration of their dynamics

Original: [Figure]
Revised: [Figure]

Original: The results, shown in Figure 2, highlight the strength of ProMP in terms of sample efﬁciency and asymptotic performance. 
Revised: The results in Figure 2 highlight the strength of ProMP in terms of sample efﬁciency and asymptotic performance. In the meta-gradient estimator study in Fig. 3, we demonstrate the positive effect of the LVC objective, as it consistently outperforms the other estimators. In contrast, DiCE learns

Original: They also demonstrate the positive effect of the LVC objective: LVC-VPG, even though optimized with vanilla policy gradient, is often able to achieve comparable results to the the prior methods that are optimized with TRPO.
Revised: [Figure] only slowly when compared to the other approaches.

Original: When compared to E-MAML-VPG, LVC proves strictly superior in  performance which underpins the soundness of the theory developed throughout this paper.
Revised: As we have motivated mathematically and substantiate empirically in the following experiment, the poor performance of DiCE may be ascribed to the high variance of its meta-gradient estimates. The fact that the results of MAML and EMAML are comparable underpins the ineffectiveness of the naive pre-update credit assignment (i [Figure] e. formulation II), as discussed in section 4.

Original: Results for four additional environments are displayed in Appendix D along with hyperparameter settings, environment speciﬁcations and a wall-clock time comparison of the algorithms.
Revised: Results for four additional environments are displayed in Appendix D along with hyperparameter settings, environment speciﬁcations and a wall-clock time comparison of the algorithms.

Original: 7.  E STIMATOR V ARIANCE AND I TS E FFECT ON M ETA -L EARNING
Revised: 7. G RADIENT E STIMATOR V ARIANCE AND I TS E FFECT ON M ETA -L EARNING

Original: [Figure] In Section 5 we discussed how the DiCE formulation yields unbiased but high variance estimates of the RL-objective hessian and served as motivation for the low variance curvature (LVC) estimator.
Revised: In Section 5 we discussed how the DiCE formulation yields unbiased but high variance estimates of the RL-objective hessian and served as motivation for the low variance curvature (LVC) estimator.

Original: Here we investigate the meta-gradient variance of both estimators as well as its implication on the learning performance.
Revised: Here we investigate the meta-gradient variance of both estimators as well as its implication on the learning performance.

Original: Speciﬁcally, we report the relative standard deviation of the meta-policy gradients as well as the average return throughout the learning process in  the HalfCheetahFwdBack environment.
Revised: Speciﬁcally, we report the relative standard deviation of the metapolicy gradients as well as the average return throughout the learning process in three of the metaenvironments.

Original: The results, depicted in Figure 3, highlight the advantage of the low variance curvature estimate.
Revised: The results, depicted in Figure 4, highlight the advantage of the low variance curvature estimate.

Original: The trajectory level dependencies inherent in the DiCE estimator lead to a metagradient standard deviation that is on average two times higher when compared to LVC.
Revised: The trajectory level dependencies inherent in the DiCE estimator leads to a meta-gradient standard deviation that is on average 60% higher when compared to LVC.

Original: As the learning curves indicate, the noisy gradients impede sample efﬁcient meta-learning in case of DiCE.
Revised: As the learning curves indicate, the noisy gradients may be a driving factor for the poor performance of DiCE, impeding sample efﬁcient meta-learning.

Original: Meta-policy search based on the LVC estimator leads to substantially better learning properties.
Revised: Meta-policy search based on the LVC estimator leads to substantially better sample-efﬁciency and asymptotic performance.

Revised: In case of HalfCheetahFwdBack, we observe some unstable learning behavior of LVC-VPG which is most likely caused by the bias of LVC in combination with the naive VPG optimizer.

Revised: However, the mechanisms in ProMP that ensure proximity w.r.t.

Revised: to the policys KL-divergence seem to counteract these instabilities during training, giving us a stable and efﬁcient meta-learning algorithm.

Original: 7. C OMPARISON OF I NITIAL S AMPLING D ISTRIBUTIONS
Revised: 7. C OMPARISON OF I NITIAL S AMPLING D ISTRIBUTIONS

Original: Here we evaluate the effect of the different objectives on the learned pre-update sampling distribution.
Revised: Here we evaluate the effect of the different objectives on the learned pre-update sampling distribution.

Original: We compare the low variance curvature (LVC) estimator with TRPO (LVC-TRPO) against MAML (Finn et al., 2017) and E-MAML-TRPO (Stadie et al., 2018) in a 2D environment on which the exploration behavior can be visualized.
Revised: We compare the low variance curvature (LVC) estimator with TRPO (LVC-TRPO) against MAML (Finn et al., 2017) and E-MAML-TRPO (Stadie et al., 2018) in a 2D environment on which the exploration behavior can be visualized.

Original: Each task of this environment corresponds to reaching a different corner location; however, the 2D agent only experiences reward when it is sufﬁciently close to the corner (translucent regions of Figure 4).
Revised: Each task of this environment corresponds to reaching a different corner location; however, the 2D agent only experiences reward when it is sufﬁciently close to the corner (translucent regions of Figure 5).

Original: Thus, to successfully identify the task, the agent must explore the different regions.
Revised: Thus, to successfully identify the task, the agent must explore the different regions.

Original: We perform three inner adaptation steps on each task, allowing the agent to fully change its behavior from exploration to exploitation.
Revised: We perform three inner adaptation steps on each task, allowing the agent to fully change its behavior from exploration to exploitation.

Original: LVC 

Revised: LVC

Original: MAML 

Revised: MAML

Original: E-MAML
Revised: E-MAML

Original: [Figure]
Revised: Pre-update Post-update

Original: The different exploration-exploitation strategies are displayed in Figure 4.
Revised: The different exploration-exploitation strategies are displayed in Figure 5.

Original: Since the MAML implementation does not assign credit to the pre-update sampling trajectory, it is unable to learn a sound exploration strategy for task identiﬁcation and thus fails to accomplish the task.
Revised: Since the MAML implementation does not assign credit to the pre-update sampling trajectory, it is unable to learn a sound exploration strategy for task identiﬁcation and thus fails to accomplish the task.

Original: On the other hand,
Revised: On the other hand,

Original: E-MAML, which corresponds to formulation II, learns to explore in long but random paths: because it can only assign credit to batches of pre-update trajectories, there is no notion of which actions in particular facilitate good task adaptation.
Revised: E-MAML, which corresponds to formulation II, learns to explore in long but random paths: because it can only assign credit to batches of pre-update trajectories, there is no notion of which actions in particular facilitate good task adaptation.

Original: As consequence the adapted policy slightly misses the task-speciﬁc target.
Revised: As consequence the adapted policy slightly misses the task-speciﬁc target.

Original: The LVC estimator, instead, learns a consistent pattern of exploration, visiting each of the four regions, which it harnesses to fully solve the task.
Revised: The LVC estimator, instead, learns a consistent pattern of exploration, visiting each of the four regions, which it harnesses to fully solve the task.

Original: 7. G RADIENT U PDATE D IRECTIONS OF THE T WO M ETA -RL F ORMULATIONS
Revised: 7. G RADIENT U PDATE D IRECTIONS OF THE T WO M ETA -RL F ORMULATIONS

Original: To shed more light on the differences of the gradients of formulation I and formulation II, we evaluate the meta-gradient updates and the corresponding convergence to the optimum of both formulations in a simple 1D environment.
Revised: To shed more light on the differences of the gradients of formulation I and formulation II, we evaluate the meta-gradient updates and the corresponding convergence to the optimum of both formulations in a simple 1D environment.

Original: In this environment, the agent starts in a random position in the real line and has to reach a goal located at the position 1 or -1.
Revised: In this environment, the agent starts in a random position in the real line and has to reach a goal located at the position 1 or -1.

Original: In order to visualize the convergence, we parameterize the policy with only two parameters θ 0 and θ 1 .
Revised: In order to visualize the convergence, we parameterize the policy with only two parameters θ 0 and θ 1 .

Original: We employ formulation I by optimizing the
Revised: We employ formulation I by optimizing the

Original: DiCE objective with VPG, and formulation II by optimizing its (E-MAML) objective with VPG.
Revised: DiCE objective with VPG, and formulation II by optimizing its (E-MAML) objective with VPG.

Original: Figure 5 depicts meta-gradient updates of the parameters θ i for both formulations.
Revised: Figure 6 depicts meta-gradient updates of the parameters θ i for both formulations.

Original: Formulation I (red) exploits the internal structure of the adaptation update yielding faster and steadier convergence to the optimum.
Revised: Formulation I (red) exploits the internal structure of the adaptation update yielding faster and steadier convergence to the optimum.

Original: Due to its inferior credit assignment, formulation II (green) produces noisier gradient estimates leading to worse convergence properties.
Revised: Due to its inferior credit assignment, formulation II (green) produces noisier gradient estimates leading to worse convergence properties.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper we propose a novel Meta-RL algorithm, proximal meta-policy search (ProMP), which fully optimizes for the pre-update sampling distribution leading to effective task identiﬁcation.
Revised: In this paper we propose a novel Meta-RL algorithm, proximal meta-policy search (ProMP), which fully optimizes for the pre-update sampling distribution leading to effective task identiﬁcation.

Original: Our method is the result of a theoretical analysis of gradient-based Meta-RL formulations, based on which we develop the low variance curvature (LVC) surrogate objective that produces low variance meta-policy gradient estimates.
Revised: Our method is the result of a theoretical analysis of gradient-based Meta-RL formulations, based on which we develop the low variance curvature (LVC) surrogate objective that produces low variance meta-policy gradient estimates.

Original: Experimental results demonstrate that our approach surpasses previous meta-reinforcement learning approaches in a diverse set of continuous control tasks.
Revised: Experimental results demonstrate that our approach surpasses previous meta-reinforcement learning approaches in a diverse set of continuous control tasks.

Original: Finally, we underpin our theoretical contributions with illustrative examples which further justify the soundness and effectiveness of our method.
Revised: Finally, we underpin our theoretical contributions with illustrative examples which further justify the soundness and effectiveness of our method.

Original: R A T WO M ETA -P OLICY G RADIENT F ORMULATIONS
Revised: A T WO M ETA -P OLICY G RADIENT F ORMULATIONS

Original: In this section we discuss two different gradient-based meta-learning formulations, derive their gradients and analyze the differences between them.
Revised: In this section we discuss two different gradient-based meta-learning formulations, derive their gradients and analyze the differences between them.

Original: A. M ETA -P OLICY G RADIENT F ORMULATION I
Revised: A. M ETA -P OLICY G RADIENT F ORMULATION I

Original: The ﬁrst meta-learning formulation, known as MAML (Finn et al., 2017), views the inner update rule
Revised: The ﬁrst meta-learning formulation, known as MAML (Finn et al., 2017), views the inner update rule

Original: U ( θ, T ) as a mapping from the pre-update parameter θ and the task T to an adapted policy parameter θ (cid:48) .
Revised: U ( θ, T ) as a mapping from the pre-update parameter θ and the task T to an adapted policy parameter θ (cid:48) .

Original: The update function can be viewed as stand-alone procedure that encapsulates sampling from the task-speciﬁc trajectory distribution P T ( τ | π θ ) and updating the policy parameters.
Revised: The update function can be viewed as stand-alone procedure that encapsulates sampling from the task-speciﬁc trajectory distribution P T ( τ | π θ ) and updating the policy parameters.

Original: Building on this concept, the meta-objective can be written as
Revised: Building on this concept, the meta-objective can be written as

Original: [Equation]
Revised: [Equation]

Original: with θ (cid:48):
Revised: with θ (cid:48)

Revised: [Equation]

Original: =
Revised: =

Original: U ( θ, T )

Original: [Equation]
Revised: [Equation]

Original: The task-speciﬁc gradients follow as
Revised: The task-speciﬁc gradients follow as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: In order to derive the gradients of the inner update ∇ θθ (cid:48)
Revised: In order to derive the gradients of the inner update ∇ θ

Original: = ∇ θ U ( θ, T ) it is necessary to know the structure of U .
Revised: [Equation] T ) it is necessary to know the structure of U .

Original: The main part of this paper assumes the inner update rule to be a policy gradient descent step
Revised: The main part of this paper assumes the inner update rule to be a policy gradient descent step

Original: [Equation]
Revised: [Equation]

Original: Thereby the second term in (19) is the local curvature (hessian) of the inner adaptation objective function.
Revised: Thereby the second term in (19) is the local curvature (hessian) of the inner adaptation objective function.

Original: The correct hessian of the inner objective can be derived as follows:
Revised: The correct hessian of the inner objective can be derived as follows:

Original: [Equation]
Revised: [Equation]

Original: A. M ETA -P OLICY G RADIENT F ORMULATION II
Revised: A. M ETA -P OLICY G RADIENT F ORMULATION II

Original: The second meta-reinforcement learning formulation views the the inner update θ (cid:48)
Revised: The second meta-reinforcement learning formulation views the the inner update θ (cid:48)

Original: =
Revised: =

Original: U ( θ, τ 1: N ) as a deterministic function of the pre-update policy parameters θ and N trajectories τ 1: N ∼
Revised: U ( θ, τ 1: N ) as a deterministic function of the pre-update policy parameters θ and N trajectories τ 1: N ∼

Original: P T ( τ 1: N | θ ) sampled from the pre-update trajectory distribution.
Revised: P T ( τ 1: N | θ ) sampled from the pre-update trajectory distribution.

Original: This formulation was introduced in Al-Shedivat et al.
Revised: This formulation was introduced in Al-Shedivat et al.

Original: and further discussed with respect to its exploration properties in Stadie et al.
Revised: and further discussed with respect to its exploration properties in Stadie et al.

Original: Viewing U as a function that adapts the policy parameters θ to a speciﬁc task T given policy rollouts in this task, the corresponding meta-learning objective can be written as
Revised: Viewing U as a function that adapts the policy parameters θ to a speciﬁc task T given policy rollouts in this task, the corresponding meta-learning objective can be written as

Original: [Equation]
Revised: [Equation]

Original:  Since the ﬁrst part of the gradient derivation is agnostic to the inner update rule U ( θ, τ 1: N ) , we only assume that the inner update function U is differentiable w.r.t.
Revised: θ [Equation] Since the ﬁrst part of the gradient derivation is agnostic to the inner update rule U ( θ, τ 1: N ) , we only assume that the inner update function U is differentiable w.r.t.

Original: θ .
Revised: θ .

Original: First we rewrite the meta-objective
Revised: First we rewrite the meta-objective

Original: J ( θ ) as expectation of task speciﬁc objectives J II T ( θ ) under the task distribution.
Revised: J ( θ ) as expectation of task speciﬁc objectives J II T ( θ ) under the task distribution.

Original: This allows us to express the meta-policy gradients as expectation of task-speciﬁc gradients:
Revised: This allows us to express the meta-policy gradients as expectation of task-speciﬁc gradients:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The task speciﬁc gradients can be calculated as follows
Revised: The task speciﬁc gradients can be calculated as follows

Original: [Equation]
Revised: [Equation]

Original: As in A.1 the structure of U ( θ, τ 1: N ) must be known in order to derive the gradient ∇ θ θ (cid:48) .
Revised: As in A.1 the structure of U ( θ, τ 1: N ) must be known in order to derive the gradient ∇ θ θ (cid:48) .

Original: Since we assume the inner update to be vanilla policy gradient, the respective gradient follows as
Revised: Since we assume the inner update to be vanilla policy gradient, the respective gradient follows as

Original: [Equation]
Revised: [Equation]

Original: ∇ θ log

Original: [Equation]

Original: The respective gradient of U ( θ, τ 1: N ) follows as
Revised: The respective gradient of U ( θ, τ 1: N ) follows as

Original: [Equation]
Revised: [Equation]

Original: A. C OMPARING THE G RADIENTS OF THE T WO F ORMULATIONS
Revised: A. C OMPARING THE G RADIENTS OF THE T WO F ORMULATIONS

Original: In the following we analyze the differences between the gradients derived for the two formulations.
Revised: In the following we analyze the differences between the gradients derived for the two formulations

Original:  To do so, we begin with ∇ θ
Revised: [Equation] To do so [Equation]

Original:  J I T ( θ ) by inserting the gradient of the inner adaptation step (19) into (17):
Revised: we begin with ∇ θ J I T ( θ ) by inserting the gradient of the inner adaptation step (19) into (17):

Original: [Equation]
Revised: [Equation]

Original: We can substitute the hessian of the inner objective by its derived expression from (24) and then rearrange the terms.
Revised: We can substitute the hessian of the inner objective by its derived expression from (24) and then rearrange the terms.

Original: Also note that ∇ θ log P T ( τ | θ ) =
Revised: Also note that ∇ θ log P T ( τ | θ ) =

Original: ∇ θ log π θ ( τ )
Revised: ∇ θ log π θ ( τ )

Original: =
Revised: =

Original: (cid:80) H − 1 t =1 log π θ ( a t | s t ) where
Revised: (cid:80) H − 1 t =1 log π θ ( a t | s t ) where

Original: H is the MDP horizon.
Revised: H is the MDP horizon.

Original: [Equation]
Revised: [Equation]

Original: Next, we rearrange the gradient of J II into a similar form as ∇ θ J I T ( θ ) .
Revised: Next, we rearrange the gradient of J II into a similar form as ∇ θ J I T ( θ ) .

Original: For that, we start by inserting (28) for ∇ θ θ (cid:48) and replacing the expectation over pre-update trajectories τ 1: N by the expectation over a single trajectory τ .
Revised: For that, we start by inserting (28) for ∇ θ θ (cid:48) and replacing the expectation over pre-update trajectories τ 1: N by the expectation over a single trajectory τ .

Original: [Equation]
Revised: [Equation]

Original: While the ﬁrst part of the gradients match ((32) and (34)), the second part ((33) and (35)) differs.
Revised: While the ﬁrst part of the gradients match ((32) and (34)), the second part ((33) and (35)) differs.

Original: Since the second gradient term can be viewed as responsible for shifting the pre-update sampling distribution P T ( τ | θ ) towards higher post-update returns, we refer to it as ∇ θ J pre ( τ , τ (cid:48) ) .
Revised: Since the second gradient term can be viewed as responsible for shifting the pre-update sampling distribution P T ( τ | θ ) towards higher post-update returns, we refer to it as ∇ θ J pre ( τ , τ (cid:48) ) .

Original: To further analyze the difference between ∇ θ J I pre and ∇ θ
Revised: To further analyze the difference between ∇ θ J I pre and ∇ θ

Original: J II pre we slightly rearrange (33) and put both gradient terms next to each other:
Revised: J II pre we slightly rearrange (33) and put both gradient terms next to each other:

Original: [Equation]
Revised: [Equation]

Original: In the following we interpret and and compare of the derived gradient terms, aiming to provide intuition for the differences between the formulations:
Revised: In the following we interpret and and compare of the derived gradient terms, aiming to provide intuition for the differences between the formulations:

Original: The ﬁrst gradient term J post that matches in both formulations corresponds to a policy gradient step on the post-update policy π θ (cid:48) .Since θ (cid:48) itself is a function of θ , the term (cid:0) I + αR [Equation] τ
Revised: The ﬁrst gradient term J post that matches in both formulations corresponds to a policy gradient step on the post-update policy π θ (cid:48) .

Original: [Equation] ∇ 2 θ log π θ [Equation] )) (cid:1) can be seen as linear transformation of the policy gradient update R ( τ (cid:48) ) ∇ θ (cid:48) log π θ ( τ (cid:48) ) from the post-update parameter θ (cid:48) into θ .
Revised: Since θ (cid:48) itself is a function of θ , the term (cid:0) I + αR ( τ ) ∇ 2 θ log π θ ( τ )) (cid:1) can be seen as linear transformation of the policy gradient update R ( τ (cid:48) ) ∇ θ (cid:48) log π θ ( τ (cid:48) ) from the post-update parameter θ (cid:48) into θ .

Original: Although J post takes into account the functional relationship between θ (cid:48) and θ , it does not take into account the pre-update sampling distribution P T ( τ | θ ) .
Revised: Although J post takes into account the functional relationship between θ (cid:48) and θ , it does not take into account the pre-update sampling distribution P T ( τ | θ ) .

Original: This is where ∇ θ J pre comes into play: ∇ θ J I pre can be viewed as policy gradient update of the preupdate policy π θ w.r.t.
Revised: This is where ∇ θ J pre comes into play: ∇ θ J I pre can be viewed as policy gradient update of the preupdate policy π θ w.r.t.

Original: to the post-update return R ( τ (cid:48) ) .
Revised: to the post-update return R ( τ (cid:48) ) .

Original: Hence this gradient term aims to shift the pre-update sampling distribution so that higher post-update returns are achieved.
Revised: Hence this gradient term aims to shift the pre-update sampling distribution so that higher post-update returns are achieved.

Original: However, ∇ θ J II pre does not take into account the causal dependence of the post-update policy on the pre-update policy.
Revised: However, ∇ θ J II pre does not take into account the causal dependence of the post-update policy on the pre-update policy.

Original: Thus a change in θ due to ∇ θ J II pre may counteract the change due to ∇ θ J II post .
Revised: Thus a change in θ due to ∇ θ J II pre may counteract the change due to ∇ θ J II post .

Original: In contrast, ∇ θ J I pre takes the dependence of the the post-update policy on the pre-update sampling distribution into account.
Revised: In contrast, ∇ θ J I pre takes the dependence of the the post-update policy on the pre-update sampling distribution into account.

Original: Instead of simply weighting the gradients of the pre-update policy ∇ θ log π θ ( τ ) with R ( τ (cid:48) )
Revised: Instead of simply weighting the gradients of the pre-update policy ∇ θ log π θ ( τ ) with R ( τ (cid:48) )

Original: as in ∇ θ
Revised: as in ∇ θ

Original: J I post , ∇ θ
Revised: J I post , ∇ θ

Original: J
Revised: J

Original: I post weights the gradients with inner product of the pre-update and post-update policy gradients.
Revised: I post weights the gradients with inner product of the pre-update and post-update policy gradients.

Original: This inner product can be written as
Revised: This inner product can be written as

Original: [Equation]
Revised: [Equation]

Original:  wherein δ denotes the angle between the the inner and outer pre-update and post-update policy gradients.
Revised: (38) wherein δ denotes the angle between the the inner and outer pre-update and post-update policy gradients.

Original: Hence, ∇ θ
Revised: Hence, ∇ θ

Original: J
Revised: J

Original: I post steers the pre-update policy towards not only towards larger post-updates returns but also towards larger adaptation steps α ∇ θ J inner , and better alignment of pre- and postupdate policy gradients.
Revised: I post steers the pre-update policy towards not only towards larger post-updates returns but also towards larger adaptation steps α ∇ θ J inner , and better alignment of pre- and postupdate policy gradients.

Original: This directly optimizes for maximal improvement / adaptation for the respective task.
Revised: This directly optimizes for maximal improvement / adaptation for the respective task.

Original: See Li et al.
Revised: See Li et al.

Original: ; Nichol et al.
Revised: ; Nichol et al.

Original: (2018) for a comparable analysis in case of domain generalization and supervised meta-learning.
Revised: (2018) for a comparable analysis in case of domain generalization and supervised meta-learning.

Original: Also note that (38) allows formulation I to perform credit assignment on the trajectory level whereas formulation II can only assign credit to entire batches of N pre-update trajectories τ 1: N .
Revised: Also note that (38) allows formulation I to perform credit assignment on the trajectory level whereas formulation II can only assign credit to entire batches of N pre-update trajectories τ 1: N .

Original: As a result, we expect the ﬁrst meta-policy gradient formulation to learn faster and more stably since the respective gradients take the dependence of the pre-update returns on the pre-update sampling distribution into account while this causal link is neglected in the second formulation.
Revised: As a result, we expect the ﬁrst meta-policy gradient formulation to learn faster and more stably since the respective gradients take the dependence of the pre-update returns on the pre-update sampling distribution into account while this causal link is neglected in the second formulation.

Original: B E STIMATING THE M ETA -P OLICY G RADIENTS
Revised: B E STIMATING THE M ETA -P OLICY G RADIENTS

Original: When employing formulation I for gradient-based meta-learning, we aim maximize the loss
Revised: When employing formulation I for gradient-based meta-learning, we aim maximize the loss

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: with θ (cid:48):
Revised: with θ (cid:48)

Original:  = θ + α ∇ θ E τ ∼ P T ( τ | θ )
Revised: [Equation] = θ + [Equation]

Original: [ R ( τ )] (39) by performing a form of gradient-descent on J ( θ ) .
Revised: by performing a form of gradient-descent on J ( θ ) .

Original: Note that we, from now on, assume J :=
Revised: Note that we, from now on, assume J :=

Original: J I and thus omit the superscript indicating the respective meta-learning formulation.
Revised: J I and thus omit the superscript indicating the respective meta-learning formulation.

Original: As shown in A.2 the gradient can be derived as ∇ θ J ( θ )
Revised: As shown in A.2 the gradient can be derived as ∇ θ J ( θ )

Original: =
Revised: =

Original: E ( T ) ∼ ρ ( T ) [ ∇ θ J T ( θ )] with ∇ [Equation]
Revised: E ( T ) ∼ ρ ( T ) [ ∇ θ J T ( θ )] with ∇ θ J T ( θ ) =

Original:  T 
Revised: E τ (cid:48) ∼ P T ( τ

Original: [Equation]
Revised: [Equation]

Original: where ∇ 2 θ J inner ( θ ) := ∇ 2 θ E τ ∼ P T ( τ | θ )
Revised: where ∇ 2 θ J inner ( θ ) := ∇ 2 θ E τ ∼ P T ( τ | θ )

Original: [ R ( τ )] denotes hessian of the inner adaptation objective w.r.t.
Revised: [ R ( τ )] denotes hessian of the inner adaptation objective w.r.t.

Original: θ .
Revised: θ .

Original: This section concerns the question of how to properly estimate this hessian.
Revised: This section concerns the question of how to properly estimate this hessian.

Original: B. E STIMATING G RADIENTS OF THE RL R EWARD O BJECTIVE
Revised: B. E STIMATING G RADIENTS OF THE RL R EWARD O BJECTIVE

Original: Since the expectation over the trajectory distribution P T ( τ | θ ) is in general intractable, the score function trick is typically used to used to produce a Monte Carlo estimate of the policy gradients.
Revised: Since the expectation over the trajectory distribution P T ( τ | θ ) is in general intractable, the score function trick is typically used to used to produce a Monte Carlo estimate of the policy gradients.

Original: Although the gradient estimate can be directly deﬁned, when using a automatic-differentiation toolbox it is usually more convenient to use an objective function whose gradients correspond to the policy gradient estimate.
Revised: Although the gradient estimate can be directly deﬁned, when using a automatic-differentiation toolbox it is usually more convenient to use an objective function whose gradients correspond to the policy gradient estimate.

Original: Due to the Policy Gradient Theorem (PGT)
Revised: Due to the Policy Gradient Theorem (PGT)

Original: Sutton et al.
Revised: Sutton et al.

Original: (2000) such a “surrogate” objective can be written as:
Revised: (2000) such a “surrogate” objective can be written as:

Original: [Equation]
Revised: [Equation]

Original: While (41) and (42) are equivalent (Peters & Schaal, 2006), the more popular formulation formulation (41) can be seen as forward looking credit assignment while (42) can be interpreted as backward looking credit assignment (Foerster et al., 2018).
Revised: While (41) and (42) are equivalent (Peters & Schaal, 2006), the more popular formulation formulation (41) can be seen as forward looking credit assignment while (42) can be interpreted as backward looking credit assignment (Foerster et al., 2018).

Original: A generalized procedure for constructing “surrogate” objectives for arbitrary stochastic computation graphs can be found in Schulman et al.
Revised: A generalized procedure for constructing “surrogate” objectives for arbitrary stochastic computation graphs can be found in Schulman et al.

Original: (2015a).
Revised: (2015a).

Original: B. A DECOMPOSITION OF THE HESSIAN
Revised: B. A DECOMPOSITION OF THE HESSIAN

Original: Estimating the the hessian of the reinforcement learning objective has been discussed in Furmston et al.
Revised: Estimating the the hessian of the reinforcement learning objective has been discussed in Furmston et al.

Original: and Baxter & Bartlett (2001) with focus on second order policy gradient methods.
Revised: and Baxter & Bartlett (2001) with focus on second order policy gradient methods.

Original: In the inﬁnite horizon MDP case, Baxter & Bartlett (2001) derive a decomposition of the hessian.
Revised: In the inﬁnite horizon MDP case, Baxter & Bartlett (2001) derive a decomposition of the hessian.

Original: In the following, we extend their ﬁnding to the ﬁnite horizon case.
Revised: In the following, we extend their ﬁnding to the ﬁnite horizon case.

Original: Proposition.
Revised: Proposition.

Original: The hessian of the RL objective can be decomposed into four matrix terms:
Revised: The hessian of the RL objective can be decomposed into four matrix terms:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Here Q π θ t 
Revised: t ) (cid:32) H

Original: [Equation]
Revised: [Equation]

Original: s t , a
Revised: 1 (cid:88) t (cid:48)

Original: [Equation]
Revised: = t r ( s t (cid:48) , a t (cid:48) ) (cid:33)(cid:35) (45)

Original: ) =
Revised: H 12 =

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: | [Equation]
Revised: T (

Original: [Equation]
Revised: [Equation]

Original: − 1 t
Revised: θ ) (cid:34)

Original: [Equation]
Revised: [Equation]

Original: r
Revised: (cid:88) t = ∇ θ log

Original: [Equation]
Revised: [Equation]

Original:  t
Revised: θ ( a t

Original: [Equation]
Revised: [Equation]

Original: ) | s t
Revised: ) ∇ θ Q π θ t ( s t

Original: [Equation]
Revised: [Equation]

Original: at
Revised: a

Original: [Equation]
Revised: [Equation]

Revised: ) (cid:62) (cid:35) (46)

Revised: Here Q π θ t ( s t , a t ) =

Revised: E τ t +1: H − 1 ∼ P T ( ·| θ ) (cid:104)(cid:80)

Original:  denotes the expected state-action value function under policy π θ at time t .
Revised: H − 1 t (cid:48) = t r ( s t (cid:48) , a t (cid:48) ) | s t , a t (cid:105) denotes the expected state-action value function under policy π θ at time t .

Original: Proof.
Revised: Proof.

Original: As derived in (24), the hessian of J inner ( θ ) follows as:
Revised: As derived in (24), the hessian of J inner ( θ ) follows as:

Original: [Equation]
Revised: [Equation]

Original: The term in (50) is equal to H 2 .
Revised: The term in (50) is equal to H 2 .

Original: We continue by showing that the remaining term in (51) is equivalent to H 

Revised: We continue by showing that the remaining term in (51) is equivalent to H

Original: + H 

Revised: + H

Original: + H (cid:62) 12 .
Revised: + H (cid:62) 12 .

Original: For that, we split the inner double sum in (51) into three components: 
Revised: For that, we split the inner double sum in (51) into three components: E [Equation] | θ

Original: [Equation]
Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: +

Revised: [Equation]

Revised: 

Revised: [Equation]

Original: By changing the backward looking summation over outer products into a forward looking summation of rewards
Revised: By changing the backward looking summation over outer products into a forward looking summation of rewards

Original: [Equation]
Revised: [Equation]

Original: (53) can be shown to be equal to H 1 :
Revised: (53) can be shown to be equal to H 1 :

Original: [Equation]
Revised: [Equation]

Original: By simply exchanging the summation indices t (cid:48) and h in (55) it is straightforward to show that (55) is the transpose of (54).
Revised: By simply exchanging the summation indices t (cid:48) and h in (55) it is straightforward to show that (55) is the transpose of (54).

Original: Hence it is sufﬁcient to show that (54) is equivalent to H 12 .
Revised: Hence it is sufﬁcient to show that (54) is equivalent to H 12 .

Original: However, instead of following the direction of the previous proof we will now start with the deﬁnition of Hand derive the expression in (54).
Revised: However, instead of following the direction of the previous proof we will now start with the deﬁnition of Hand derive the expression in (54).

Original: [Equation]
Revised: [Equation]

Original: The gradient of Q π θ t can be expressed recursively:
Revised: The gradient of Q π θ t can be expressed recursively:

Original: [Equation]
Revised: [Equation]

Original: By induction, it follows that
Revised: By induction, it follows that

Original: [Equation]
Revised: [Equation]

Revised: When inserting (63) into (59) and swapping the summation

Original: [Equation]
Revised: [Equation]

Original: When inserting (63) into (59) and swapping the summation, we are able to show that H 12 is equivalent to (54).
Revised: we are able to show that H 12 is equivalent to (54).

Original: [Equation]
Revised: [Equation]

Original: This concludes the proof that the hessian of the expected sum of rewards under policy π θ and an
Revised: This concludes the proof that the hessian of the expected sum of rewards under policy π θ and an

Original: MDP with ﬁnite time horizon H can be decomposed into H 1 + H 2 + H 12 + H (cid:62) 12 .
Revised: MDP with ﬁnite time horizon H can be decomposed into H 1 + H 2 + H 12 + H (cid:62) 12 .

Original: [Equation]
Revised: [Equation]

Original: B. E STIMATING  H ESSIAN OF THE RL R EWARD O BJECTIVE
Revised: B. E STIMATING THE H ESSIAN OF THE RL R EWARD O BJECTIVE

Original: As pointed out by Al-Shedivat et al.
Revised: As pointed out by Al-Shedivat et al.

Original: ; Stadie et al.
Revised: ; Stadie et al.

Original: (2018) and Foerster et al.
Revised: (2018) and Foerster et al.

Original: (2018), simply differentiating through the gradient of surrogate objective J PGT as done in the original MAML version (Finn et al., 2017) leads to biased hessian estimates.
Revised: (2018), simply differentiating through the gradient of surrogate objective J PGT as done in the original MAML version (Finn et al., 2017) leads to biased hessian estimates.

Original: Speciﬁcally, when compared with the unbiased estimate, as derived in (24) and decomposed in Appendix B.2, both H 1 and H 

Revised: Speciﬁcally, when compared with the unbiased estimate, as derived in (24) and decomposed in Appendix B.2, both H 1 and H

Original: + H (cid:62) 12 are missing.
Revised: + H (cid:62) 12 are missing.

Original: Thus, ∇ θ J pre does not appear in the gradients of the meta-objective (i.e. ∇ θ J = ∇ θ J post ).
Revised: Thus, ∇ θ J pre does not appear in the gradients of the meta-objective (i.e. ∇ θ J = ∇ θ J post ).

Original: Only performing gradient descent with ∇ θ J post entirely neglects inﬂuences of the pre-update sampling distribution.
Revised: Only performing gradient descent with ∇ θ J post entirely neglects inﬂuences of the pre-update sampling distribution.

Original: This issue was overseen in the RL-MAML implementation of Finn et al.
Revised: This issue was overseen in the RL-MAML implementation of Finn et al.

Original: As discussed in Stadie et al.
Revised: As discussed in Stadie et al.

Original: this leads to poor performance in meta-learning problems that require exploration during the pre-update sampling.
Revised: this leads to poor performance in meta-learning problems that require exploration during the pre-update sampling.

Original: B.3. T HE DICE M ONTE -C ARLO E STIMATOR
Revised: B.3. T HE DICE M ONTE -C ARLO E STIMATOR

Original: Addressing the issue of incorrect higher-order derivatives of monte-carlo estimators, Foerster et al.
Revised: Addressing the issue of incorrect higher-order derivatives of monte-carlo estimators, Foerster et al.

Original: (2018) propose DICE which mainly builds upon an newly introduced MagicBox( (cid:3) ) operator.
Revised: (2018) propose DICE which mainly builds upon an newly introduced MagicBox( (cid:3) ) operator.

Original: This operator allows to formulate monte-carlo estimators with correct higher-order derivatives.
Revised: This operator allows to formulate monte-carlo estimators with correct higher-order derivatives.

Original: A DICE formulation of a policy gradient estimator reads as:
Revised: A DICE formulation of a policy gradient estimator reads as:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: In that, ⊥ denotes a ”stop gradient” operator (i.e. ⊥ ( f θ ( x ))
Revised: In that, ⊥ denotes a “stop gradient” operator (i.e. ⊥ ( f θ ( x ))

Original: → f θ ( x ) but ∇ θ ⊥ ( f θ ( x )) → 0 ).
Revised: → f θ ( x ) but ∇ θ ⊥ ( f θ ( x )) → 0 ).

Original: Note that → denotes a ”evaluates to” and does not necessarily imply equality w.r.t.
Revised: Note that → denotes a “evaluates to” and does not necessarily imply equality w.r.t.

Original: to gradients.
Revised: to gradients.

Original: Hence, J DICE ( θ ) evaluates to the sum of rewards at 0th order but produces the unbiased gradients ∇ nθ J DICE ( θ ) when differentiated n-times (see Foerster et al.
Revised: Hence, J DICE ( θ ) evaluates to the sum of rewards at 0th order but produces the unbiased gradients ∇ nθ J DICE ( θ ) when differentiated n-times (see Foerster et al.

Original: (2018) for proof).
Revised: (2018) for proof).

Original: To shed more light on the maverick DICE formulation, we rewrite (67) as follows:
Revised: To shed more light on the maverick DICE formulation, we rewrite (67) as follows:

Original: [Equation]
Revised: [Equation]

Original: Interpreting this novel formulation, the MagicBox operator (cid:3) θ ( { a t (cid:48) ≤ t } ) can be understood as ”dry” importance sampling weight.
Revised: Interpreting this novel formulation, the MagicBox operator (cid:3) θ ( { a t (cid:48) ≤ t } ) can be understood as “dry” importance sampling weight.

Original: At 0th order it evaluates to 1 and leaves the objective function unaffected, but when differentiated once it yields an estimator for the marginal rate of return due to a change in the policy-implied trajectory distribution.
Revised: At 0th order it evaluates to 1 and leaves the objective function unaffected, but when differentiated once it yields an estimator for the marginal rate of return due to a change in the policy-implied trajectory distribution.

Original: In the following we show that on expectation 1) the gradients of (81) match standard policy gradients and 2) its hessian estimate is equal to the hessian of inner RL objective, derived in B.2.
Revised: In the following we show that on expectation 1) the gradients of (81) match standard policy gradients and 2) its hessian estimate is equal to the hessian of inner RL objective, derived in B.2.

Original: [Equation]
Revised: [Equation]

Original: Here, (71) corresponds to the backward looking credit assignment formulation of policy gradients ∇ θ J PGT as discussed in B.1.
Revised: Here, (71) corresponds to the backward looking credit assignment formulation of policy gradients ∇ θ J PGT as discussed in B.1.

Original: Once again we take the derivative in order to obtain the Hessian of
Revised: Once again we take the derivative in order to obtain the Hessian of

Original: J DICE :
Revised: J DICE :

Original: [Equation]
Revised: [Equation]

Original: In expectation E τ ∼ P T ( τ | θ ) [ ∇ 2 θ J DICE ] the DICE monte carlo estimate of the hessian is equivalent to the hessian of the inner objective.
Revised: In expectation, E τ ∼ P T ( τ | θ ) [ ∇ 2 θ J DICE ] the DICE monte carlo estimate of the hessian is equivalent to the hessian of the inner objective.

Original: To show this, we use the expression of ∇ 2 θ J inner (49):
Revised: To show this, we use the expression of ∇ 2 θ J inner (49):

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: B. B IAS AND VARIANCE OF THE CURVATURE ESTIMATE
Revised: B. B IAS AND VARIANCE OF THE CURVATURE ESTIMATE

Original: As shown in the previous section, ∇ 2 θ J DICE provides an unbiased estimate of the hessian of the inner objective J inner =
Revised: As shown in the previous section, ∇ 2 θ J DICE provides an unbiased estimate of the hessian of the inner objective J inner =

Original: E τ ∼ P T ( τ | θ )
Revised: E τ ∼ P T ( τ | θ )

Original: [ R ( τ )] .
Revised: [ R ( τ )] .

Original: However, recall the DICE objective involves a product of importance weights along the trajectory.
Revised: However, recall the DICE objective involves a product of importance weights along the trajectory.

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Taking the 2nd derivative of this product leads to the outer product of sums in (74) which is of high variance w.r.t to τ .
Revised: Taking the 2nd derivative of this product leads to the outer product of sums in (74) which is of high variance w.r.t to τ .

Original: Speciﬁcally, this outer product of sums can be decomposed into three terms
Revised: Speciﬁcally, this outer product of sums can be decomposed into three terms

Original: H 

Revised: H

Original: + H 

Revised: + H

Original: + H (cid:62) 12 (see Appendix B.2).
Revised: + H (cid:62) 12 (see Appendix B.2).

Original: As noted by Furmston et al.
Revised: As noted by Furmston et al.

Original: , H 12 + H (cid:62) 12 is particularly difﬁcult to estimate.
Revised: , H 12 + H (cid:62) 12 is particularly difﬁcult to estimate.

Original: In section 7.2 we empirically show that the high variance curvature estimates obtained with the DICE objective require large batch sizes and impede sample efﬁcient learning.
Revised: In section 7.2 we empirically show that the high variance curvature estimates obtained with the DICE objective require large batch sizes and impede sample efﬁcient learning.

Original: In the following we develop a low variance curvature (LVC) estimator J LVC which matches J DICE at the gradient level and yields lower-variance estimates of the hessian by neglecting H 

Revised: In the following we develop a low variance curvature (LVC) estimator J LVC which matches J DICE at the gradient level and yields lower-variance estimates of the hessian by neglecting H

Original: + H (cid:62) 12 .
Revised: + H (cid:62) 12 .

Original: Before formally introducing J LVC , we motivate such estimator starting with the policy gradient estimate that was originally derived in Sutton et al.
Revised: Before formally introducing J LVC , we motivate such estimator starting with the policy gradient estimate that was originally derived in Sutton et al.

Original: (2000), followed by marginalizing the trajectory level distribution P T ( τ | θ ) over states s t and actions a t .
Revised: (2000), followed by marginalizing the trajectory level distribution P T ( τ | θ ) over states s t and actions a t .

Original: Note that we omit reward baselines for notational simplicity.
Revised: Note that we omit reward baselines for notational simplicity.

Original: [Equation]
Revised: [Equation]

Original: In that, p π θ t ( s t ) denotes the state visitation frequency at time step t , i.e. the probability density of being in s t after t steps under the policy π θ .
Revised: In that, p π θ t ( s t ) denotes the state visitation frequency at time step t , i.e. the probability density of being in s t after t steps under the policy π θ .

Original: In the general case p π θ t ( s t ) is intractable but depends on the policy parameter θ .
Revised: In the general case p π θ t ( s t ) is intractable but depends on the policy parameter θ .

Original: We make the simplifying assumption that p π θ t ( s t ) is ﬁxed in a local region of θ .
Revised: We make the simplifying assumption that p π θ t ( s t ) is ﬁxed in a local region of θ .

Original: Since we make this assumption at the gradient level, this corresponds to a 1st order
Revised: Since we make this assumption at the gradient level, this corresponds to a 1st order

Original: Taylor expansion of p π θ t ( s t ) in θ .
Revised: Taylor expansion of p π θ t ( s t ) in θ .

Original: Note that this assumption is also used in the Monotonic Policy
Revised: Note that this assumption is also used in the Monotonic Policy

Original: Improvement Theory (Kakade & Langford, 2002; Schulman et al., 2015a).
Revised: Improvement Theory (Kakade & Langford, 2002; Schulman et al., 2015a).

Original: Based on this condition, the hessian follows as derivative of (83) whereby a ”stop gradient” expression around the state visitation frequency p π θ t ( s t ) resembles the 1st order Taylor approximation:
Revised: Based on this condition, the hessian follows as derivative of (83) whereby a “stop gradient” expression around the state visitation frequency p π θ t ( s t ) resembles the 1st order Taylor approximation

Original: [Equation]
Revised: [Equation]

Original: Since the expectation in (84) is intractable it must be evaluated by a monte carlo estimate
Revised: Since the expectation in (84) is intractable it must be evaluated by a monte carlo estimate.

Original: [Equation] However [Equation] simply replacing the expectation with an average of samples trajectories induces a wrong hessian that does not correspond to (86) since outer product of log-gradients would be missing when differentiated.
Revised: However, simply replacing the expectation with an average of samples trajectories induces a wrong hessian that does not correspond to (86) since outer product of log-gradients would be missing when differentiated.

Original: To ensure that automatic differentiation still yields the correct hessian [Equation] we add a ”dry” importance weight comparable to DICE:[Equation]
Revised: To ensure that automatic differentiation still yields the correct hessian, we add a “dry” importance weight comparable to DICE:

Original: [Equation]
Revised: [Equation]

Original: When integrated this resembles the LVC ”surrogate” objective J LVC .
Revised: When integrated this resembles the LVC “surrogate” objective J LVC .

Original: [Equation]
Revised: [Equation]

Original: The gradients of J LVC match ∇ θ J DICE and resemble an unbiased policy gradient estimate:
Revised: The gradients of J LVC match ∇ θ J DICE and resemble an unbiased policy gradient estimate:

Original: [Equation]
Revised: [Equation]

Original: The respective Hessian can be obtained by differentiating (90):
Revised: The respective Hessian can be obtained by differentiating (90):

Original: [Equation]
Revised: [Equation]

Original: In expectation ∇ 2 θ J LVC is equivalent to H 

Revised: In expectation ∇ 2 θ J LVC is equivalent to H

Original: + H 2 :
Revised: + H 2 :

Original: [Equation]
Revised: [Equation]

Original: The Hessian ∇ 2 θ J LVC no longer provides an unbiased estimate of ∇ 2 θ J inner since neglects the matrix term H 

Revised: The Hessian ∇ 2 θ J LVC no longer provides an unbiased estimate of ∇ 2 θ J inner since neglects the matrix term H

Original: + H (cid:62) 12 .
Revised: + H (cid:62) 12 .

Original: This approximation is based on the assumption that the state visitation distribution is locally unaffected by marginal changes in θ and leads to a substantial reduction of variance in the hessian estimate.
Revised: This approximation is based on the assumption that the state visitation distribution is locally unaffected by marginal changes in θ and leads to a substantial reduction of variance in the hessian estimate.

Original: Furmston et al.
Revised: Furmston et al.

Original: (2016) show that under certain conditions (i.e. inﬁnite horizon
Revised: (2016) show that under certain conditions (i.e. inﬁnite horizon

Original: MDP, sufﬁciently rich policy parameterisation) the term H 

Revised: MDP, sufﬁciently rich policy parameterisation) the term H

Original: + H (cid:62) 12 vanishes around a local optimum θ ∗ .
Revised: + H (cid:62) 12 vanishes around a local optimum θ ∗ .

Original: Given that the conditions hold, this implies that E τ [ ∇ 2 θ J LVC ]
Revised: Given that the conditions hold, this implies that E τ [ ∇ 2 θ J LVC ]

Original: → E τ [ ∇ 2 θ J DICE ] as θ → θ ∗ , i.e. the bias of the LCV estimator becomes negligible close to the local optimum.
Revised: → E τ [ ∇ 2 θ J DICE ] as θ → θ ∗ , i.e. the bias of the LCV estimator becomes negligible close to the local optimum.

Original: The experiments in section 7.2 conﬁrm this theoretical argument empirically and show that using the low variance curvature estimates obtained through J LVC improve the sample-efﬁciency of meta-learning by a signiﬁcant margin.
Revised: The experiments in section 7.2 conﬁrm this theoretical argument empirically and show that using the low variance curvature estimates obtained through J LVC improve the sample-efﬁciency of meta-learning by a signiﬁcant margin.

Original: C P ROXIMAL P OLICY S EARCH M ETHODS C. M ONOTONIC P OLICY I MPROVEMENT T HEORY
Revised: C P ROXIMAL P OLICY S EARCH M ETHODS C. M ONOTONIC P OLICY I MPROVEMENT T HEORY

Original: In this section provides a brief introduction to policy performance bounds and the theory of monotonic policy improvement in the setting of reinforcement learning.
Revised: This section provides a brief introduction to policy performance bounds and the theory of monotonic policy improvement in the setting of reinforcement learning.

Original: While Section 6 discusses the extension of this theory to meta learning, the following explanations assume a standard RL setting where T is exogenously given.
Revised: While Section 6 discusses the extension of this theory to meta learning, the following explanations assume a standard RL setting where T is exogenously given.

Original: Hence, we will omit mentioning  dependence on T for notational brevity.
Revised: Hence, we will omit mentioning the dependence on T for notational brevity.

Original: Since  monotonic policy improvement frameworks relies on inﬁnite-time horizion MDPs, we assume H → ∞ for the remainder of this chapter.
Revised: Since the monotonic policy improvement frameworks relies on inﬁnite-time horizon MDPs, we assume H → ∞ for the remainder of this chapter.

Original: In addition to the expected reward J ( π ) under policy π , we will use the state value function V π , the state-action value function Q π as well as the advantage function A π :[Equation] −
Revised: In addition to the expected reward J ( π ) under policy π , we will use the state value function V π , the state-action value function Q π as well as the advantage function A π :

Original: [Equation]
Revised: [Equation]

Original: with a t ∼ π ( a t | s t ) and s t +1 ∼ p ( s t +1 | s t , a t ) .
Revised: with a t ∼ π ( a t | s t ) and s t +1 ∼ p ( s t +1 | s t , a t ) .

Original: The expected return under a policy ˜ π can be expressed as the sum of the expected return of another policy π and the expected discounted advantage of ˜ π over π (see Schulman et al.
Revised: The expected return under a policy ˜ π can be expressed as the sum of the expected return of another policy π and the expected discounted advantage of ˜ π over π (see Schulman et al.

Original: (2015a) for proof).
Revised: (2015a) for proof).

Original: [Equation]
Revised: [Equation]

Original: Let d π denote the discounted state visitation frequency:
Revised: Let d π denote the discounted state visitation frequency:

Original: [Equation]
Revised: [Equation]

Original: We can use d π to express the expectation over trajectories τ ∼ p π ( τ ) in terms of states and actions:
Revised: We can use d π to express the expectation over trajectories τ ∼ p π ( τ ) in terms of states and actions:

Original: [Equation]
Revised: [Equation]

Original: Local policy search aims to ﬁnd a policy update π → ˜ π in the proximity of π so that J (˜ π ) is maximized.
Revised: Local policy search aims to ﬁnd a policy update π → ˜ π in the proximity of π so that J (˜ π ) is maximized.

Original: Since J ( π ) is not affected by the policy update π → ˜ π , it is sufﬁcient to maximize the expected advantage under ˜ π .
Revised: Since J ( π ) is not affected by the policy update π → ˜ π , it is sufﬁcient to maximize the expected advantage under ˜ π .

Original: However, the complex dependence of d ˜ π ( s ) on ˜ π makes it hard to directly maximize the objective in (102).
Revised: However, the complex dependence of d ˜ π ( s ) on ˜ π makes it hard to directly maximize the objective in (102).

Original: Using a local approximation of (102) where it is assumed that the state visitation frequencies d π and d ˜ π are identical, the optimization can be phrased as
Revised: Using a local approximation of (102) where it is assumed that the state visitation frequencies d π and d ˜ π are identical, the optimization can be phrased as

Original: [Equation]
Revised: [Equation]

Original: In the following we refer to ˜ J
Revised: In the following we refer to ˜ J

Original: (˜ π ) as surrogate objective.
Revised: (˜ π ) as surrogate objective.

Original: It can be shown that the surrogate objective ˜
Revised: It can be shown that the surrogate objective ˜

Original: J matches J to ﬁrst order when π = ˜
Revised: J matches J to ﬁrst order when π = ˜

Original: π (see Kakade & Langford (2002)).
Revised: π (see Kakade & Langford (2002)).

Original: If π θ is a parametric and differentiable function with parameter vector θ , this means that for any θ o :
Revised: If π θ is a parametric and differentiable function with parameter vector θ , this means that for any θ o :

Original: [Equation]
Revised: [Equation]

Original: When π (cid:54) =
Revised: When π (cid:54) =

Original: ˜ π , an approximation error of the surrogate objective ˜ J w.r.t.
Revised: ˜ π , an approximation error of the surrogate objective ˜ J w.r.t.

Original: to the true objective J is introduced.
Revised: to the true objective J is introduced.

Original: Achiam et al.
Revised: Achiam et al.

Original: (2017) derive a lower bound for the true expected return of ˜ π :
Revised: (2017) derive a lower bound for the true expected return of ˜ π :

Original: [Equation]
Revised: [Equation]

Original: ˜ π || π ] (105) with C = √ 
Revised: ] (105) with C = √ 2 γ 1 − γ max s | E

Original: [Equation]
Revised: [Equation]

Original: γ
Revised: ∼

Original: [Equation]
Revised: [Equation]

Original: max
Revised: ( a

Original: [Equation]
Revised: [Equation]

Original: | E [Equation] [Equation]
Revised: [ A

Original: [Equation]
Revised: [Equation]

Original: ] |
Revised: )] |

Original: C.
Revised: C.

Original: T RUST R EGION P OLICY O PTIMIZATION (TRPO)
Revised: T RUST R EGION P OLICY O PTIMIZATION (TRPO)

Original: Trust region policy optimization (TPRO) (Schulman et al., 2015a) attempts to approximate the bound in (105) by phrasing local policy search as a constrained optimization problem:[Equation]
Revised: Trust region policy optimization (TPRO) (Schulman et al., 2015a) attempts to approximate the bound in (105) by phrasing local policy search as a constrained optimization problem:

Original: [Equation]
Revised: [Equation]

Original: Thereby the KL-constraint δ induces a local trust region around the current policy π θ o .
Revised: Thereby the KL-constraint δ induces a local trust region around the current policy π θ o .

Original: A practical implementation of TPRO uses a quadratic approximation of the KL-constraint which leads to the following update rule:
Revised: A practical implementation of TPRO uses a quadratic approximation of the KL-constraint which leads to the following update rule:

Original: [Equation]
Revised: [Equation]

Original: with g := ∇ θ E s ∼ d πθo ( s ) a ∼ π θo ( a | s ) (cid:104)
Revised: with g := ∇ θ E s ∼ d πθo ( s ) a ∼ π θo ( a | s ) (cid:104)

Original: π θ ( a | s ) π θo ( a | s )
Revised: π θ ( a | s ) π θo ( a | s )

Original: A π θo ( s , a ) (cid:105) being the gradient of the objective and F = ∇ 2 θ ¯ D KL [ π θ o ||
Revised: A π θo ( s , a ) (cid:105) being the gradient of the objective and F = ∇ 2 θ ¯ D KL [ π θ o ||

Original: π θ ] the Fisher information matrix of the current policy π θ o .
Revised: π θ ] the Fisher information matrix of the current policy π θ o .

Original: In order to avoid the cubic time complexity that arise when inverting F , the Conjugate Gradient (CG) algorithm is typically used to approximate the Hessian vector product F − 1 g .
Revised: In order to avoid the cubic time complexity that arise when inverting F , the Conjugate Gradient (CG) algorithm is typically used to approximate the Hessian vector product F − 1 g .

Original: C. P ROXIMAL P OLICY O PTIMIZATION (PPO)
Revised: C. P ROXIMAL P OLICY O PTIMIZATION (PPO)

Original: While TPRO is framed as constrained optimization, the theory discussed in Appendix C.1 suggest to optimize the lower bound.
Revised: While TPRO is framed as constrained optimization, the theory discussed in Appendix C.1 suggest to optimize the lower bound.

Original: Based on this insight, Schulman et al.
Revised: Based on this insight, Schulman et al.

Original: (2017) propose adding a KL penalty to the objective and solve the following unconstrained optimization problem:
Revised: (2017) propose adding a KL penalty to the objective and solve the following unconstrained optimization problem:

Original: [Equation]
Revised: [Equation]

Original: However, they also show that it is not sufﬁcient to set a ﬁxed penalty coefﬁcient β and propose two alternative methods, known as Proximal Policy Optimization (PPO) that aim towards alleviating this issue:
Revised: However, they also show that it is not sufﬁcient to set a ﬁxed penalty coefﬁcient β and propose two alternative methods, known as Proximal Policy Optimization (PPO) that aim towards alleviating this issue:

Original: 1) Adapting the KL coefﬁcient β so that a desired target KL-divergence ¯ D KL [ π θ o ||
Revised: 1) Adapting the KL coefﬁcient β so that a desired target KL-divergence ¯ D KL [ π θ o ||

Original: π θ ] between the policy before and after the parameter update is achieved 2)
Revised: π θ ] between the policy before and after the parameter update is achieved 2)

Original: Clipping the likelihood ratio so that the optimization has no incentive to move the policy π θ too far away from the original policy π θ o .
Revised: Clipping the likelihood ratio so that the optimization has no incentive to move the policy π θ too far away from the original policy π θ o .

Original: A corresponding optimization objective reads as:
Revised: A corresponding optimization objective reads as:

Original: [Equation]
Revised: [Equation]

Original: Empirical results show that the latter approach leads to better learning performance (Schulman et al., 2017).
Revised: Empirical results show that the latter approach leads to better learning performance (Schulman et al., 2017).

Original: Since PPO objective keeps π θ in proximity of π θ o , it allows to perform multiple gradient steps without re-sampling trajectories from the updated policy.
Revised: Since PPO objective keeps π θ in proximity of π θ o , it allows to perform multiple gradient steps without re-sampling trajectories from the updated policy.

Original: This property substantially improves the data-efﬁciency of PPO over vanilla policy gradient methods which need to re-estimate the gradients after each step.
Revised: This property substantially improves the data-efﬁciency of PPO over vanilla policy gradient methods which need to re-estimate the gradients after each step.

Original: D E XPERIMENTS D. H YPERPARAMETER C HOICE
Revised: D E XPERIMENTS D. H YPERPARAMETER C HOICE

Original: The optimal hyperparameter for each algorithm was determined using parameter sweeps.
Revised: The optimal hyperparameter for each algorithm was determined using parameter sweeps.

Original: Tablecontains the hyperparameter settings used for the different algorithms.
Revised: Tablecontains the hyperparameter settings used for the different algorithms.

Original: Any environment speciﬁc modiﬁcations are noted in the respective paragraph describing the environment.
Revised: Any environment speciﬁc modiﬁcations are noted in the respective paragraph describing the environment.

Original: D. E NVIRONMENT S PECIFICATIONS
Revised: D. E NVIRONMENT S PECIFICATIONS

Original: PointEnv (used in the experiment in 7.3)
Revised: PointEnv (used in the experiment in 7.3)

Original: • Trajectory Length :• Num Adapt Steps :
Revised: • Trajectory Length :• Num Adapt Steps :

Original: All

Original: [Table]
Revised: [Table]

Original: In this environment, each task corresponds to one corner of the area.
Revised: In this environment, each task corresponds to one corner of the area.

Original: The point mass must reach the goal by applying directional forces.
Revised: The point mass must reach the goal by applying directional forces.

Original: The agent only experiences a reward when within a certain radius of the goal, and the magnitude of the reward is equal to the distance to the goal.
Revised: The agent only experiences a reward when within a certain radius of the goal, and the magnitude of the reward is equal to the distance to the goal.

Original: HalfCheetahFwdBack, AntFwdBack, WalkerFwdBack, HumanoidFwdBack
Revised: HalfCheetahFwdBack, AntFwdBack, WalkerFwdBack, HumanoidFwdBack

Original: • Trajectory Length : 100 (HalfCheetah, Ant); 200 (Humanoid, Walker) • Num Adapt Steps:
Revised: • Trajectory Length : 100 (HalfCheetah, Ant); 200 (Humanoid, Walker) • Num Adapt Steps:

Original: The task is chosen between two directions - forward and backward.
Revised: The task is chosen between two directions - forward and backward.

Original: Each agent must run along the goal direction as far as possible, with reward equal to average velocity minus control costs.
Revised: Each agent must run along the goal direction as far as possible, with reward equal to average velocity minus control costs.

Original: AntRandDirec, HumanoidRandDirec
Revised: AntRandDirec, HumanoidRandDirec

Original: • Trajectory Length : 100 (Ant); 200 (Humanoid) • Num Adapt Steps:
Revised: • Trajectory Length : 100 (Ant); 200 (Humanoid) • Num Adapt Steps:

Original: Each task corresponds to a random direction in the XY plane.
Revised: Each task corresponds to a random direction in the XY plane.

Original: As above, each agent must learn to run in that direction as far as possible, with reward equal to average velocity minus control costs.
Revised: As above, each agent must learn to run in that direction as far as possible, with reward equal to average velocity minus control costs.

Original: AntRandGoal
Revised: AntRandGoal

Original: • Trajectory Length :• Num Adapt Steps:
Revised: • Trajectory Length :• Num Adapt Steps:

Original: In this environment, each task is a location randomly chosen from a circle in the XY plane.
Revised: In this environment, each task is a location randomly chosen from a circle in the XY plane.

Original: The goal is not given to the agent - it must learn to locate, approach, and stop at the target.
Revised: The goal is not given to the agent - it must learn to locate, approach, and stop at the target.

Original: The agent receives a penalty equal to the distance from the goal.
Revised: The agent receives a penalty equal to the distance from the goal.

Original: HopperRandParams, WalkerRandParams
Revised: HopperRandParams, WalkerRandParams

Original: • Trajectory Length :• Inner LR : 0.
Revised: • Trajectory Length :• Inner LR : 0.

Original: • Num Adapt Steps:
Revised: • Num Adapt Steps:

Original: The agent must move forward as quickly as it can.
Revised: The agent must move forward as quickly as it can.

Original: Each task is a different randomization of the simulation parameters, including friction, joint mass, and inertia.
Revised: Each task is a different randomization of the simulation parameters, including friction, joint mass, and inertia.

Original: The agent receives a reward equal to its velocity.
Revised: The agent receives a reward equal to its velocity.

Original: D. F URTHER E XPERIMENTS R ESULTS
Revised: D. F URTHER E XPERIMENTS R ESULTS

Original: In addition to the six environments displayed in 2, we ran experiments on the other four continuous control environments described above.
Revised: In addition to the six environments displayed in 2, we ran experiments on the other four continuous control environments described above.

Original: The results are displayed in 6.
Revised: The results are displayed in 7.

Original: [Figure]
Revised: [Figure]

Original: A v e r a g e r e t u r n
Revised: v e r a g e r e t u r n

Original: AntFwdBack

Original: [Figure]
Revised: [Figure]

Original: In addition to the improved sample complexity and better asymptotic performance, another advantage of ProMP is its computation time.
Revised: In addition to the improved sample complexity and better asymptotic performance, another advantage of ProMP is its computation time.

Original: Figure 7 shows the average time spent per iteration throughout the learning process in the humanoid environment differences of ProMP, LVC-VPG, and MAML-TRPO.
Revised: Figure 8 shows the average time spent per iteration throughout the learning process in the humanoid environment differences of ProMP, LVC-VPG, and MAML-TRPO.

Original: Due to the expensive conjugate gradient steps used in TRPO, MAML takes far longer than either ﬁrst order method.
Revised: Due to the expensive conjugate gradient steps used in TRPO, MAML takes far longer than either ﬁrst order method.

Original: Since ProMP takes multiple stochastic gradient descent steps per iteration, it leads to longer outer update times compared to VPG, but in both cases the update time is a fraction of the time spent sampling from the environment.
Revised: Since ProMP takes multiple stochastic gradient descent steps per iteration, it leads to longer outer update times compared to VPG, but in both cases the update time is a fraction of the time spent sampling from the environment.

Original: The difference in sampling time is due to the reset process: resetting the environment when the agent “dies” is an expensive operation.
Revised: The difference in sampling time is due to the reset process: resetting the environment when the agent “dies” is an expensive operation.

Original: ProMP acquires better performance quicker, and as a result the agent experiences longer trajectories and the environment is reset less often.
Revised: ProMP acquires better performance quicker, and as a result the agent experiences longer trajectories and the environment is reset less often.

Original: In our setup, instances of the environment are run in parallel and performing a reset blocks all environments.
Revised: In our setup, instances of the environment are run in parallel and performing a reset blocks all environments.

Original: [Figure]
Revised: [Figure]