Original: N O DICE: A N I NVESTIGATION
Revised: N O DICE: A N I NVESTIGATION

Original: OF

Original: THE
Revised: THE

Original: B IAS - V ARIANCE T RADEOFF IN M ETA -G RADIENTS
Revised: B IASV ARIANCE T RADEOFF IN M ETA -G RADIENTS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Meta-gradients provide a general approach for optimizing the meta-parameters of reinforcement learning (RL) algorithms.
Revised: Meta-gradients provide a general approach for optimizing the meta-parameters of reinforcement learning (RL) algorithms.

Original: Estimation of meta-gradients is central to the performance of these meta-algorithms, and has been studied in the setting of MAML-style short-horizon meta-RL problems.
Revised: Estimation of meta-gradients is central to the performance of these meta-algorithms, and has been studied in the setting of MAML-style short-horizon meta-RL problems.

Original: In this context, prior work has investigated the estimation of the Hessian of the RL objective, as well as tackling the problem of credit assignment to pre-adaptation behavior by making a sampling correction .
Revised: In this context, prior work has investigated the estimation of the Hessian of the RL objective, as well as tackling the problem of credit assignment to pre-adaptation behavior by making a sampling correction .

Original: However, we show that Hessian estimation, implemented for example by DiCE and its variants, always add bias and can also add variance to metagradient estimation.
Revised: However, we show that Hessian estimation, implemented for example by DiCE and its variants, always add bias and can also add variance to metagradient estimation.

Original: DiCE-like approaches are therefore unlikely to lie on Pareto frontier of the bias-variance tradeoff and should not be pursued in the context of meta-gradients for RL.
Revised: DiCE-like approaches are therefore unlikely to lie on Pareto frontier of the bias-variance tradeoff and should not be pursued in the context of meta-gradients for RL.

Original: Meanwhile, the sampling correction has not been studied in the important long-horizon setting, where the inner optimization trajectories must be truncated for computational tractability.
Revised: Meanwhile, the sampling correction has not been studied in the important long-horizon setting, where the inner optimization trajectories must be truncated for computational tractability.

Original: We study the bias and variance tradeoff induced by truncated backpropagation in combination with a weighted sampling correction.
Revised: We study the bias and variance tradeoff induced by truncated backpropagation in combination with a weighted sampling correction.

Original: While prior work has implicitly chosen points in this biasvariance space, we disentangle the sources of bias and variance and present an empirical study which relates existing estimators to each other.
Revised: While prior work has implicitly chosen points in this biasvariance space, we disentangle the sources of bias and variance and present an empirical study which relates existing estimators to each other.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recently, meta-gradients have been used for adapting the hyperparameters of state-of-the-art policy gradient RL algorithms online (Flennerhag et al., 2021), as well as for learning black-box RL algorithms from scratch (Oh et al., 2020).
Revised: Recently, meta-gradients have been used for adapting the hyperparameters of state-of-the-art policy gradient RL algorithms online (Flennerhag et al., 2021), as well as for learning black-box RL algorithms from scratch (Oh et al., 2020).

Original: The estimation of meta-gradients is central to the performance of these algorithms, but has received little focused study.
Revised: The estimation of meta-gradients is central to the performance of these algorithms, but has received little focused study.

Original: Deriving unbiased estimators of meta-gradients is more involved than for standard policy gradients because the meta-parameters affect the data distribution used for computing the meta-gradient.
Revised: Deriving unbiased estimators of meta-gradients is more involved than for standard policy gradients because the meta-parameters affect the data distribution used for computing the meta-gradient.

Original: Al-Shedivat et al.
Revised: Al-Shedivat et al.

Original: (2017) present an unbiased meta-gradient estimator that correctly accounts for the changes in the data distribution, by making a sampling correction which is added to the direct meta-gradient.
Revised: (2017) present an unbiased meta-gradient estimator that correctly accounts for the changes in the data distribution, by making a sampling correction which is added to the direct meta-gradient.

Original: However, their derivation and experiments are limited to MAML-style meta-RL algorithms (Finn et al., 2017), which learn the initialization of a policy such that it may achieve good performance after a small number of policy gradient updates.
Revised: However, their derivation and experiments are limited to MAML-style meta-RL algorithms (Finn et al., 2017), which learn the initialization of a policy such that it may achieve good performance after a small number of policy gradient updates.

Original: Further work studying meta-gradient estimation for meta-RL is also largely restricted to this context.
Revised: Further work studying meta-gradient estimation for meta-RL is also largely restricted to this context.

Original: Computing the meta-gradient requires differentiating through the agent’s parameter update, which may itself include an estimated policy gradient.
Revised: Computing the meta-gradient requires differentiating through the agent’s parameter update, which may itself include an estimated policy gradient.

Original: The correct meta-gradient of a standard policy gradient update can be computed by straightforward backpropagation of a surrogate loss, as is typical in RL.
Revised: The correct meta-gradient of a standard policy gradient update can be computed by straightforward backpropagation of a surrogate loss, as is typical in RL.

Original: However, Foerster et al.
Revised: However, Foerster et al.

Original: (2018) and a number of subsequent works (Rothfuss et al., 2018;
Revised: (2018) and a number of subsequent works (Rothfuss et al., 2018;

Original: Liu et al., 2019; Farquhar et al., 2019; Mao et al., 2019; Tang et al., 2021) have claimed that an estimate of the expected policy Hessian should be included in the meta-gradient estimator, and provide methods for calculating such an estimate or reducing its variance.
Revised: Liu et al., 2019; Farquhar et al., 2019; Mao et al., 2019; Tang et al., 2021) have claimed that an estimate of the expected policy Hessian should be included in the meta-gradient estimator, and provide methods for calculating such an estimate or reducing its variance.

Original: In this work, we present the surprising result that using an estimate of the expected Hessian in the meta-gradient estimator actually adds bias, which can degrade performance.
Revised: In this work, we present the surprising result that using an estimate of the expected Hessian in the meta-gradient estimator actually adds bias, which can degrade performance.

Original: Such a term could only be used if the agent made an expected policy gradient update, rather than the sampled update which must be taken using ﬁnite data.
Revised: Such a term could only be used if the agent made an expected policy gradient update, rather than the sampled update which must be taken using ﬁnite data.

Original: This focus on estimating expected Hessians may have distracted the community from the real challenges in meta-gradient estimation that arise in the long-horizon setting, where the sampling correction has not been addressed.
Revised: This focus on estimating expected Hessians may have distracted the community from the real challenges in meta-gradient estimation that arise in the long-horizon setting, where the sampling correction has not been addressed.

Original: In the short-horizon setting, where the sampling correction has been previously explored, the metalearning algorithm can afford to consider the whole optimization trajectory of the inner learning problem.
Revised: In the short-horizon setting, where the sampling correction has been previously explored, the metalearning algorithm can afford to consider the whole optimization trajectory of the inner learning problem.

Original: Due to computational constraints, in a long-horizon setting, the inner loop must be truncated to a small fraction of the full optimization trajectory.
Revised: Due to computational constraints, in a long-horizon setting, the inner loop must be truncated to a small fraction of the full optimization trajectory.

Original: As a result, neither the direct metagradient nor the sampling correction may be calculated exactly.
Revised: As a result, neither the direct metagradient nor the sampling correction may be calculated exactly.

Original: It is therefore unclear how the truncated sampling correction affects the bias and variance of the meta-gradient estimation.
Revised: It is therefore unclear how the truncated sampling correction affects the bias and variance of the meta-gradient estimation.

Original: Previous works (Xu et al., 2018; 2020; Oh et al., 2020) have simply opted for truncated meta-gradient estimators without sampling correction, without mention of this potential source of bias.
Revised: Previous works (Xu et al., 2018; 2020; Oh et al., 2020) have simply opted for truncated meta-gradient estimators without sampling correction, without mention of this potential source of bias.

Original: While impressive results have been achieved with these estimators, they are biased due to both truncation and the lack of sampling correction.
Revised: While impressive results have been achieved with these estimators, they are biased due to both truncation and the lack of sampling correction.

Original: In this paper, we study the bias-variance tradeoff arising from these two approximations, and how bias and variance may be traded off by varying the truncation horizon and the weight given to the sampling correction.
Revised: In this paper, we study the bias-variance tradeoff arising from these two approximations, and how bias and variance may be traded off by varying the truncation horizon and the weight given to the sampling correction.

Original: We ﬁnd that while the sampling correction does not uniformly reduce the bias of the truncated estimators, it nevertheless results in convergence to better local optima.
Revised: We ﬁnd that while the sampling correction does not uniformly reduce the bias of the truncated estimators, it nevertheless results in convergence to better local optima.

Original: The cost of using the sampling correction is increased variance.
Revised: The cost of using the sampling correction is increased variance.

Original: Our contributions can be summarized as follows.
Revised: Our contributions can be summarized as follows.

Original: (1) We show mathematically why using the expected policy Hessian estimator is in error, and demonstrate that the resulting bias harms performance in practice.
Revised: (1) We show mathematically why using the expected policy Hessian estimator is in error, and demonstrate that the resulting bias harms performance in practice.

Original: (2) We characterize the bias-variance tradeoff due to a truncated optimization horizon and the sampling correction in an empirical study, relating existing approaches to each other and showing how to interpolate between them.
Revised: (2) We characterize the bias-variance tradeoff due to a truncated optimization horizon and the sampling correction in an empirical study, relating existing approaches to each other and showing how to interpolate between them.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: We assume that the RL algorithm is learning a control policy in a Markov decision process (MDP) deﬁned by a tuple ( S , A , p 0 , p, r, γ ) , where S is the space of states, A the space of actions, p 0 ( s 0 ) the distribution over initial states, p ( s t +1 | s t , a t ) the conditional probability distribution of the next state given a state and an action, r ( s t , a t ) the reward function for the transition, and γ ∈ (0 , 1] a discount factor.
Revised: We assume that the RL algorithm is learning a control policy in a Markov decision process (MDP) deﬁned by a tuple ( S , A , p 0 , p, r, γ ) , where S is the space of states, A the space of actions, p 0 ( s 0 ) the distribution over initial states, p ( s t +1 | s t , a t ) the conditional probability distribution of the next state given a state and an action, r ( s t , a t ) the reward function for the transition, and γ ∈ (0 , 1] a discount factor.

Original: A trajectory τ = ( s 0 , a 0 , s 1 , a 1 , . . .
Revised: A trajectory τ = ( s 0 , a 0 , s 1 , a 1 , . . .

Original: , s H , a H ) ∈ T , is a sequence of states and actions of length H sampled from the dynamics deﬁned by the MDP and the policy, where the horizon H may be inﬁnite.
Revised: , s H , a H ) ∈ T , is a sequence of states and actions of length H sampled from the dynamics deﬁned by the MDP and the policy, where the horizon H may be inﬁnite.

Original: T is the space of trajectories.
Revised: T is the space of trajectories.

Original: The probability of sampling τ is given by
Revised: The probability of sampling τ is given by

Original: [Equation]
Revised: [Equation]

Original: where π ( a t | s t ; θ ) is the policy parametrized by θ ∈ R d θ .
Revised: where π ( a t | s t ; θ ) is the policy parametrized by θ ∈ R d θ .

Original: The objective of an RL algorithm is to train a policy, that maximizes the expected discounted sum of rewards when interacting with the environment
Revised: The objective of an RL algorithm is to train a policy, that maximizes the expected discounted sum of rewards when interacting with the environment

Original: [Equation]
Revised: [Equation]

Original: For brevity we write R ( τ ) =
Revised: For brevity we write R ( τ ) =

Original: (cid:80)
Revised: (cid:80)

Original: Ht
Revised: Ht

Original: =

Revised: =

Original: γ t r ( s t , a t ) , where s t and a t come from τ .
Revised: γ t r ( s t , a t ) , where s t and a t come from τ .

Original: In our derivations we use the Monte Carlo return R ( τ ) for simplicity, but in practical algorithms it is usually replaced by an estimated return or advantage to reduce variance.
Revised: In our derivations we use the Monte Carlo return R ( τ ) for simplicity, but in practical algorithms it is usually replaced by an estimated return or advantage to reduce variance.

Original: A policy gradient algorithm updates the parameter by gradient ascent on the policy gradient given by
Revised: A policy gradient algorithm updates the parameter by gradient ascent on the policy gradient given by

Original: [Equation]
Revised: [Equation]

Original: Typically, the expectation in equation 3 cannot be evaluated exactly and is instead approximated from samples by
Revised: Typically, the expectation in equation 3 cannot be evaluated exactly and is instead approximated from samples by

Original: [Equation]
Revised: [Equation]

Original:  where D ∈ T N is a tuple of N trajectories collected with the policy parametrized by θ also known as a batch.
Revised: ∈D [Equation] where D ∈ T N is a tuple of N trajectories collected with the policy parametrized by θ also known as a batch.

Original: The probability of the batch is p ( D ; θ ) =
Revised: The probability of the batch is p ( D ; θ ) =

Original: (cid:81) τ ∈D p ( τ ; θ ) .
Revised: (cid:81) τ ∈D p ( τ ; θ ) .

Original: Meta-gradients come into play in the meta-learning problem of optimizing the parameters η of an update function Ψ( η, θ i , D i ) :
Revised: Meta-gradients come into play in the meta-learning problem of optimizing the parameters η of an update function Ψ( η, θ i , D i ) :

Original: R d η × R d θ × T N → R d θ , which computes an updated set of parameters for a policy.
Revised: R d η × R d θ × T N → R d θ , which computes an updated set of parameters for a policy.

Original: For example, a simple update function is
Revised: For example, a simple update function is

Original: [Equation]
Revised: [Equation]

Original: where the meta-parameter is the learning rate of a stochastic policy gradient update.
Revised: where the meta-parameter is the learning rate of a stochastic policy gradient update.

Original: In general, the meta-parameters can appear in any term in the update function.
Revised: In general, the meta-parameters can appear in any term in the update function.

Original: The update functions we consider in this paper use policy gradients to update the parameters but other types of update functions may be considered as well.
Revised: The update functions we consider in this paper use policy gradients to update the parameters but other types of update functions may be considered as well.

Original: Meta-gradients are often computed in a setting where the policy is updated
Revised: Meta-gradients are often computed in a setting where the policy is updated

Original: K times using the update function and the meta-parameters are optimized to maximize return with each of the K + 1 policies.
Revised: K times using the update function and the meta-parameters are optimized to maximize return with each of the K + 1 policies.

Original: The objective for such K -step meta-gradient is given by
Revised: The objective for such K -step meta-gradient is given by

Original: [Equation]
Revised: [Equation]

Original: where the outer expectation is taken over data sampled with all of the policies along the update trajectory because each θ k depends on all of the previous data through the update function.
Revised: where the outer expectation is taken over data sampled with all of the policies along the update trajectory because each θ k depends on all of the previous data through the update function.

Original: Note that by setting the returns of the trajectories for k < K equal to zero, this objective generalizes the
Revised: Note that by setting the returns of the trajectories for k < K equal to zero, this objective generalizes the

Original: MAML-style objective, which only considers the return of the ﬁnal K th policy.
Revised: MAML-style objective, which only considers the return of the ﬁnal K th policy.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: The estimation of meta-gradients has been studied in the context of short-horizon multi-task metaRL algorithms.
Revised: The estimation of meta-gradients has been studied in the context of short-horizon multi-task metaRL algorithms.

Original: In MAML (Finn et al., 2017), the initialization of an agent is learned such that its performance after a few policy gradient updates is maximized.
Revised: In MAML (Finn et al., 2017), the initialization of an agent is learned such that its performance after a few policy gradient updates is maximized.

Original: Al-Shedivat et al.
Revised: Al-Shedivat et al.

Original: (2017), derive an unbiased meta-gradient estimator for learning the initialization and extend it to a continuous adaptation setting.
Revised: (2017), derive an unbiased meta-gradient estimator for learning the initialization and extend it to a continuous adaptation setting.

Original: A similar derivation is followed by Stadie et al.
Revised: A similar derivation is followed by Stadie et al.

Original: (2018), where the improved exploratory behavior of the initial policy in the MAML-algorithm is considered additional motivation for using the unbiased estimator.
Revised: (2018), where the improved exploratory behavior of the initial policy in the MAML-algorithm is considered additional motivation for using the unbiased estimator.

Original: Fallah et al.
Revised: Fallah et al.

Original: (2020) study the convergence properties of the unbiased metagradient estimator.
Revised: (2020) study the convergence properties of the unbiased metagradient estimator.

Original: Learning the initialization necessitates considering the full optimization horizon.
Revised: Learning the initialization necessitates considering the full optimization horizon.

Original: In contrast, in the truncated setting which we also consider in this paper, the last batch of experience in a truncation window has no special signiﬁcance over the earlier ones.
Revised: In contrast, in the truncated setting which we also consider in this paper, the last batch of experience in a truncation window has no special signiﬁcance over the earlier ones.

Original: Therefore, instead of optimizing the returns at the end of the optimization trajectory as is done in MAML, it makes sense to optimize the returns throughout the lifetime of the agent as done by Antoniou et al.
Revised: Therefore, instead of optimizing the returns at the end of the optimization trajectory as is done in MAML, it makes sense to optimize the returns throughout the lifetime of the agent as done by Antoniou et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: (2016); Andrychowicz et al.
Revised: (2016); Andrychowicz et al.

Original: Many estimators of the expected Hessian have been developed and applied primarily in the MAML framework.
Revised: Many estimators of the expected Hessian have been developed and applied primarily in the MAML framework.

Original: DiCE (Foerster et al., 2018) shows that backpropagating through a stochastic policy gradient update does not result in an estimator of the expected Hessian and proposes a modiﬁed surrogate objective, which gives such estimates via backpropagation.
Revised: DiCE (Foerster et al., 2018) shows that backpropagating through a stochastic policy gradient update does not result in an estimator of the expected Hessian and proposes a modiﬁed surrogate objective, which gives such estimates via backpropagation.

Original: ProMP (Rothfuss et al., 2018) further develops the Hessian estimation strategy and derives a lower-variance estimate.
Revised: ProMP (Rothfuss et al., 2018) further develops the Hessian estimation strategy and derives a lower-variance estimate.

Original: Loaded
Revised: Loaded

Original: DiCE (Farquhar et al., 2019) derives a new expression of the DiCE objective compatible with value function approximators and introduces a hyperparameter for trading off bias and variance.
Revised: DiCE (Farquhar et al., 2019) derives a new expression of the DiCE objective compatible with value function approximators and introduces a hyperparameter for trading off bias and variance.

Original: Liu et al.
Revised: Liu et al.

Original: and Mao et al.
Revised: and Mao et al.

Original: (2019) introduce control variates to reduce the variance of the expected
Revised: (2019) introduce control variates to reduce the variance of the expected

Original: Hessian estimators.
Revised: Hessian estimators.

Original: Tang et al.
Revised: Tang et al.

Original: (2021) propose a unifying framework for the Hessian estimation problem from the perspective of off-policy evaluation.
Revised: (2021) propose a unifying framework for the Hessian estimation problem from the perspective of off-policy evaluation.

Original: All of these works consider applications of their Hessian estimators to the MAML-style meta-RL setting.
Revised: All of these works consider applications of their Hessian estimators to the MAML-style meta-RL setting.

Original: However, we show that besides having high variance, these Hessian estimators add bias to the meta-gradient estimate.
Revised: However, we show that besides having high variance, these Hessian estimators add bias to the meta-gradient estimate.

Original: Other parts of the learning algorithms have been optimized via meta-gradients besides the initialization.
Revised: Other parts of the learning algorithms have been optimized via meta-gradients besides the initialization.

Original: The objective function of the policy is learned by MetaGenRL (Kirsch et al., 2019) for a
Revised: The objective function of the policy is learned by MetaGenRL (Kirsch et al., 2019) for a

Original: DDPG-like (Lillicrap et al., 2015) algorithm.
Revised: DDPG-like (Lillicrap et al., 2015) algorithm.

Original: LPG (Oh et al., 2020) learns the full objective function for an actor-critic-style learner.
Revised: LPG (Oh et al., 2020) learns the full objective function for an actor-critic-style learner.

Original: MODAC (Veeriah et al., 2021) uses meta-gradients for option discovery.
Revised: MODAC (Veeriah et al., 2021) uses meta-gradients for option discovery.

Original: Zheng et al.
Revised: Zheng et al.

Original: (2020) learn intrinsic rewards with meta-gradients.
Revised: (2020) learn intrinsic rewards with meta-gradients.

Original: Meta-gradients have also been used to improve the performance of deep RL algorithms in the single-task setting.
Revised: Meta-gradients have also been used to improve the performance of deep RL algorithms in the single-task setting.

Original: Xu et al.
Revised: Xu et al.

Original: (2018) tune the parameters of return estimates in A2C with a meta-gradient.
Revised: (2018) tune the parameters of return estimates in A2C with a meta-gradient.

Original: STAC (Zahavy et al., 2020) follows a similar idea applying it to the hyperparameters of Impala (Espeholt et al.,2018).
Revised: STAC (Zahavy et al., 2020) follows a similar idea applying it to the hyperparameters of Impala (Espeholt et al.,

Original: BMG (Flennerhag et al., 2021) uses the same meta-parameterization as Zahavy et al.
Revised: BMG (Flennerhag et al., 2021) uses the same meta-parameterization as Zahavy et al.

Original: but improves the meta-gradient estimation to achieve the state-of-the-art model-free performance on
Revised: but improves the meta-gradient estimation to achieve the state-of-the-art model-free performance on

Original: Atari.
Revised: Atari.

Original: Our investigations are directed at improving meta-gradient in the truncated meta-optimization setting, which is used by these algorithms.
Revised: Our investigations are directed at improving meta-gradient in the truncated meta-optimization setting, which is used by these algorithms.

Original: Outside of reinforcement learning, meta-optimization has been an active area of study as well.
Revised: Outside of reinforcement learning, meta-optimization has been an active area of study as well.

Original: Wu et al.
Revised: Wu et al.

Original: (2018) focus on the bias from a short optimization horizon.
Revised: (2018) focus on the bias from a short optimization horizon.

Original: Metz et al.
Revised: Metz et al.

Original: (2019) investigate the optimization landscape of learned optimizers in supervised learning.
Revised: (2019) investigate the optimization landscape of learned optimizers in supervised learning.

Original: For a small number of metaparameters, forward-mode gradient computation can be used to overcome the memory constraint due to the long optimization horizon Franceschi et al.
Revised: For a small number of metaparameters, forward-mode gradient computation can be used to overcome the memory constraint due to the long optimization horizon Franceschi et al.

Original: These lessons from supervised learning apply to RL as well and along with our ﬁndings should be taken into consideration when using meta-gradients.
Revised: These lessons from supervised learning apply to RL as well and along with our ﬁndings should be taken into consideration when using meta-gradients.

Original:  R EVIEW OF UNBIASED META - GRADIENTS
Revised:  R EVIEW OF UNBIASED META - GRADIENTS

Original: A general unbiased meta-gradient estimator for a K -step sequence of updates was ﬁrst shown by
Revised: A general unbiased meta-gradient estimator for a K -step sequence of updates was ﬁrst shown by

Original: Al-Shedivat et al.
Revised: Al-Shedivat et al.

Original: (2017); we present a similar derivation here for convenience.
Revised: (2017); we present a similar derivation here for convenience.

Original: For a sequence of
Revised: For a sequence of

Original: K parameter updates the K th parameter is given by
Revised: K parameter updates the K th parameter is given by

Original: [Equation]
Revised: [Equation]

Original: where D i are data sampled from the environment.
Revised: where D i are data sampled from the environment.

Original: Because the D i are random variables, the parameters θ i for i > 0 are random variables too.
Revised: Because the D i are random variables, the parameters θ i for i > 0 are random variables too.

Original: They depend on the initial parameters θ 0 , the meta-parameter η , and the data.
Revised: They depend on the initial parameters θ 0 , the meta-parameter η , and the data.

Original: The meta-gradient estimator for the objective in equation 6 on the sequence of updates in equation 7 can be derived as follows:
Revised: The meta-gradient estimator for the objective in equation 6 on the sequence of updates in equation 7 can be derived as follows:

Original: [Equation]
Revised: [Equation]

Original: sampling correction
Revised: sampling correction

Original: [Equation]

Original: direct

Original: meta gradient

Original: [Equation]
Revised: [Equation]

Original: where the ∇ η θ k ∈ R d η × d θ denotes the derivative of the sequence of updates resulting in policy parameters θ k with respect to the meta-parameters.
Revised: where the ∇ η θ k ∈ R d η × d θ denotes the derivative of the sequence of updates resulting in policy parameters θ k with respect to the meta-parameters.

Revised: A more detailed derivation is provided in Appendix A.

Original: Following from the additive sequence of updates in equation 7, ∇ η θ k further expands as
Revised: Following from the additive sequence of updates in equation 7, ∇ η θ k further expands as

Original: [Equation]
Revised: [Equation]

Original: where ∇ η θ 0 can be non-zero when the initial parameters are being meta-learned, as is the case for
Revised: where ∇ η θ 0 can be non-zero when the initial parameters are being meta-learned, as is the case for

Original: MAML.
Revised: MAML.

Original: The ∇ η θ k ∇ θ k log p ( τ | θ k ) terms in equation 10 are the product of the derivative of the k th policy parameter with respect to the meta-parameters and the standard policy gradient.
Revised: The ∇ η θ k ∇ θ k log p ( τ | θ k ) terms in equation 8 are the product of the derivative of the k th policy parameter with respect to the meta-parameters and the standard policy gradient.

Original: The terms of the sum (cid:80) k −j =

Revised: The terms of the sum (cid:80) k −j =

Original: ∇ η θ j ∇ θ j log p
Revised: ∇ η θ j ∇ θ j log p

Original: ( D j | θ j ) give the sampling correction that assigns credit from the experience collected with the updated policy directly to the earlier policies.
Revised: ( D j | θ j ) give the sampling correction that assigns credit from the experience collected with the updated policy directly to the earlier policies.

Original: These latter terms are missing from the original MAML derivation (Finn et al., 2017) as shown by Al-Shedivat et al
Revised: These latter terms are missing from the original MAML derivation (Finn et al., 2017) as shown by Al-Shedivat et al.

Revised: They are also omitted in most other meta-gradient algorithms (Xu et al., 2018;

Revised: Oh et al., 2020; Zheng et al., 2018; Flennerhag et al., 2021).

Original: 4. V ARIANCE DUE TO THE SAMPLING CORRECTION
Revised: 4. V ARIANCE DUE TO THE SAMPLING CORRECTION

Original: The sampling correction terms are of the form ∇ η θ j ∇ θ j log p ( D j | θ j ) R ( τ ) , where log p ( D j | θ j )
Revised: The sampling correction terms are of the form ∇ η θ j ∇ θ j log p ( D j | θ j ) R ( τ ) , where log p ( D j | θ j )

Original: = (cid:80) τ ∈D
Revised: = (cid:80) τ ∈D

Original: j log p ( τ | θ j ) .
Revised: j log p ( τ | θ j ) .

Original: The sum over the batch elements results from the fact that every trajectory in the batch is upstream of the reward in the unrolled computation graph.
Revised: The sum over the batch elements results from the fact that every trajectory in the batch is upstream of the reward in the unrolled computation graph.

Original: In normal RL, only the samples from a given trajectory are upstream of a particular reward.
Revised: In normal RL, only the samples from a given trajectory are upstream of a particular reward.

Original: Nonetheless, a discount factor is typically used to downweight the credit assigned to temporally distant actions, reducing variance but introducing bias with respect to the undiscounted objective.
Revised: Nonetheless, a discount factor is typically used to downweight the credit assigned to temporally distant actions, reducing variance but introducing bias with respect to the undiscounted objective.

Original: Similarly, we may downweight the sampling correction terms to trade off bias and variance in meta-gradient meta-RL.
Revised: Similarly, we may downweight the sampling correction terms to trade off bias and variance in meta-gradient meta-RL.

Original: We ﬁnd that this is important in practice due to the counterintuitive growth of the variance of these terms with the inner-loop batch size.
Revised: We ﬁnd that this is important in practice due to the counterintuitive growth of the variance of these terms with the inner-loop batch size.

Original: Note that the variance of the meta-gradient estimate will of course still be reduced by increasing the outer-loop’s meta-batch size.
Revised: Note that the variance of the meta-gradient estimate will of course still be reduced by increasing the outer-loop’s meta-batch size.

Original: To enable this bias-variance tradeoff for the sampling correction terms, we use a coefﬁcient λ .
Revised: To enable this bias-variance tradeoff for the sampling correction terms, we use a coefﬁcient λ .

Original: The meta-gradient estimator estimated from samples is given by
Revised: The meta-gradient estimator estimated from samples is given by

Original: J

Original: [Equation]
Revised: [Equation]

Original: The sampling correction coefﬁcient is related to the λ hyperparameter of E-MAML (Stadie et al., 2018), but we do not separately divide by the batch size, which is done in implementations of EMAML though not motivated in the paper.
Revised: The sampling correction coefﬁcient is related to the λ hyperparameter of E-MAML (Stadie et al., 2018), but we do not separately divide by the batch size, which is done in implementations of EMAML though not motivated in the paper.

Original: Therefore, the meta-gradient estimator is unbiased when our λ = 1 . 0 .
Revised: Therefore, the meta-gradient estimator is unbiased when our λ = 1 . 0 .

Original: We investigate the bias-variance tradeoff induced by this λ in Section 6.
Revised: We investigate the bias-variance tradeoff induced by this λ in Section 6.

Original: 4. M ETA - GRADIENT ESTIMATORS FOR THE EXPECTED POLICY GRADIENT
Revised: 4. M ETA - GRADIENT ESTIMATORS FOR THE EXPECTED POLICY GRADIENT

Original: An expression for the meta-gradient without the sampling correction, used by most meta-gradient algorithms, can be derived by substituting the stochastic policy gradient in the update function with the expected policy gradient given by equation 3.
Revised: An expression for the meta-gradient without the sampling correction, used by most meta-gradient algorithms, can be derived by substituting the stochastic policy gradient in the update function with the expected policy gradient given by equation 3.

Original: In that case, there is no dependency between the policy θ k and the data sampled with earlier policies because when the policy is updated using the expected policy gradient, θ k becomes a deterministic function of η and θ 0 .
Revised: In that case, there is no dependency between the policy θ k and the data sampled with earlier policies because when the policy is updated using the expected policy gradient, θ k becomes a deterministic function of η and θ 0 .

Original: Therefore, we no longer get the sampling correction terms that consider how η affects the distribution of data.
Revised: Therefore, we no longer get the sampling correction terms that consider how η affects the distribution of data.

Original: The meta-gradient for the expected update case is given by
Revised: The meta-gradient for the expected update case is given by

Original: [Equation]
Revised: [Equation]

Original: Algorithms that estimate this by computing inner-loop updates and the meta-gradient from samples are biased because they ignore the sampling correction terms.
Revised: Algorithms that estimate this by computing inner-loop updates and the meta-gradient from samples are biased because they ignore the sampling correction terms.

Original:  E STIMATORS FOR THE EXPECTED POLICY H ESSIAN
Revised:  E STIMATORS FOR THE EXPECTED POLICY H ESSIAN

Original: Often, computing an unbiased meta-gradient estimator requires computing second order derivatives of the inner loop objective.
Revised: Often, computing an unbiased meta-gradient estimator requires computing second order derivatives of the inner loop objective.

Original: As pointed out by Foerster et al.
Revised: As pointed out by Foerster et al.

Original: (2018), computing the second order derivatives of an expected policy gradient objective is a more involved process than the corresponding computation for a sampled policy gradient objective.
Revised: (2018), computing the second order derivatives of an expected policy gradient objective is a more involved process than the corresponding computation for a sampled policy gradient objective.

Original: In this section, we look at estimators for second order derivatives of policy gradient objectives.
Revised: In this section, we look at estimators for second order derivatives of policy gradient objectives.

Original: Assuming that the update function uses some form of policy gradient, the term ∇ θ k Ψ( η, θ k , D k ) in equation 11 includes the Hessian of the policy gradient objective.
Revised: Assuming that the update function uses some form of policy gradient, the term ∇ θ k Ψ( η, θ k , D k ) in equation 9 includes the Hessian of the policy gradient objective.

Original: Using the stochastic policy gradient in the update function makes it a deterministic function of its inputs, for which the correct Hessian is easy to compute with reverse mode automatic differentiation offered by machine learning packages (e.g., (Paszke et al., 2019; Bradbury et al., 2018)) by backpropagating through the computation graph resulting from the sequence of updates given by equation 7.
Revised: Using the stochastic policy gradient in the update function makes it a deterministic function of its inputs, for which the correct Hessian is easy to compute with reverse mode automatic differentiation offered by machine learning packages (e.g., (Paszke et al., 2019; Bradbury et al., 2018)) by backpropagating through the computation graph resulting from the sequence of updates given by equation 7.

Original: However, assuming (typically incorrectly) that the inner update uses the expected policy gradient, computing an unbiased estimator of the meta-gradient requires more work.
Revised: However, assuming (typically incorrectly) that the inner update uses the expected policy gradient, computing an unbiased estimator of the meta-gradient requires more work.

Original: Developing unbiased estimators of the Hessian of the expected policy gradient has been a popular research direction within meta-RL (Rothfuss et al., 2018; Liu et al., 2019; Farquhar et al., 2019; Mao et al., 2019; Tang et al., 2021).
Revised: Developing unbiased estimators of the Hessian of the expected policy gradient has been a popular research direction within meta-RL (Rothfuss et al., 2018; Liu et al., 2019; Farquhar et al., 2019; Mao et al., 2019; Tang et al., 2021).

Original: As we show next, inclusion of the Hessian of the expected policy gradient in the meta-gradient estimator is an involved process with a questionable payoff as it leads to bias in practice and requires changes to how the update function itself is computed.
Revised: As we show next, inclusion of the Hessian of the expected policy gradient in the meta-gradient estimator is an involved process with a questionable payoff as it leads to bias in practice and requires changes to how the update function itself is computed.

Original: The expected Hessian of the policy gradient objective in equation 2 can be derived as follows
Revised: The expected Hessian of the policy gradient objective in equation 2 can be derived as follows

Original: [Equation]
Revised: [Equation]

Original: Estimating the expectation from samples we are left with the two terms
Revised: Estimating the expectation from samples we are left with the two terms

Original: [Equation]
Revised: [Equation]

Original: In contrast, when the update function uses the stochastic policy gradient estimate given by 4, the
Revised: In contrast, when the update function uses the stochastic policy gradient estimate given by 4, the

Original: Hessian is the following
Revised: Hessian is the following

Original: [Equation]
Revised: [Equation]

Original: The term ∇θ log π
Revised: The term ∇

Original: [Equation]
Revised: [Equation]

Original: ∇ θ log π( τ
Revised: log π

Original: [Equation]
Revised: [Equation]

Original: θ [Equation] (cid:62) R ( τ ) ∈ R d θ × d θ does not appear in this stochastic Hessian.
Revised: d θ does not appear in this stochastic Hessian.

Original: Although including estimates of the expected Hessian is motivated by removing bias, in the typical case where the update function uses a stochastic policy gradient, it actually adds bias because the true stochastic Hessian does not have the extra term.
Revised: Although including estimates of the expected Hessian is motivated by removing bias, in the typical case where the update function uses a stochastic policy gradient, it actually adds bias because the true stochastic Hessian does not have the extra term.

Original: It is also known to have high variance (Rothfuss et al., 2018).
Revised: It is also known to have high variance (Rothfuss et al., 2018).

Original: Consequently, we argue that developing meta-gradient algorithms based on them is not the best use of resources.
Revised: Consequently, we argue that developing meta-gradient algorithms based on them is not the best use of resources.

Original: We will illustrate the bias and variance resulting from using this estimator in practice in Section 7.
Revised: We will illustrate the bias and variance resulting from using this estimator in practice in Section 7.

Original:  S AMPLING CORRECTIONS IN A TRUNCATED OPTIMIZATION SETTING
Revised:  S AMPLING CORRECTIONS IN A TRUNCATED OPTIMIZATION SETTING

Original: Meta-gradients can be used in practice for optimizing meta-parameters of large scale deep RL algorithms.
Revised: Meta-gradients can be used in practice for optimizing meta-parameters of large scale deep RL algorithms.

Original: In these algorithms, the policy learning may take tens of thousands of update steps to converge.
Revised: In these algorithms, the policy learning may take tens of thousands of update steps to converge.

Original: Unfortunately, due to memory constraints, constructing a computation graph containing the data and parameters quickly becomes infeasible as the update horizon grows.
Revised: Unfortunately, due to memory constraints, constructing a computation graph containing the data and parameters quickly becomes infeasible as the update horizon grows.

Original: To reduce the memory requirement it is common to truncate the update horizon, which introduces bias.
Revised: To reduce the memory requirement it is common to truncate the update horizon, which introduces bias.

Original: This approach has been employed successfully in many practical meta-learning algorithms, e.g., (Xu et al., 2018;
Revised: This approach has been employed successfully in many practical meta-learning algorithms, e.g., (Xu et al., 2018;

Original: Zahavy et al., 2020;
Revised: Zahavy et al., 2020;

Original: Oh et al., 2020).
Revised: Oh et al., 2020).

Original: We know that an unbiased meta-gradient estimator can be computed using the sampling correction in the untruncated setting.
Revised: We know that an unbiased meta-gradient estimator can be computed using the sampling correction in the untruncated setting.

Original: To the best of our knowledge none of the algorithms using truncated backpropagation employ any form of the sampling correction, and it is not known how the sampling correction works in the truncated setting.
Revised: To the best of our knowledge none of the algorithms using truncated backpropagation employ any form of the sampling correction, and it is not known how the sampling correction works in the truncated setting.

Original: The truncated meta-gradient estimator is commonly implemented by approximating the estimator in equation 12 by sampling a window shorter than K update steps and limiting backpropagation to that window.
Revised: The truncated meta-gradient estimator is commonly implemented by approximating the estimator in equation 10 by sampling a window shorter than K update steps and limiting backpropagation to that window.

Original: Truncating the backpropagation changes the derivatives of the meta-parameters in equation 11, making this estimator biased.
Revised: Truncating the backpropagation changes the derivatives of the meta-parameters in equation 9, making this estimator biased.

Original: Since the sampling correction terms are a product of the derivative of the meta-parameters in equation 11 and the gradient of the log probability of a batch, they also become biased when the estimate of the derivative of the meta-parameters is biased.
Revised: Since the sampling correction terms are a product of the derivative of the meta-parameters in equation 9 and the gradient of the log probability of a batch, they also become biased when the estimate of the derivative of the meta-parameters is biased.

Original: Because of the complex dynamics of the optimization process, we have no reason to believe that the bias from truncating the backpropagation decreases monotonically with the truncation length.
Revised: Because of the complex dynamics of the optimization process, we have no reason to believe that the bias from truncating the backpropagation decreases monotonically with the truncation length.

Original: Therefore, it is possible that using the sampling correction on a truncated meta-gradient estimator increases rather than decreases the bias.
Revised: Therefore, it is possible that using the sampling correction on a truncated meta-gradient estimator increases rather than decreases the bias.

Original: In addition to changing the derivatives, approximation of the sum of all of the terms in equationwith a sum of the terms in a shorter window causes bias by modifying the surrogate loss even before backpropagation.
Revised: In addition to changing the derivatives, approximation of the sum of all of the terms in equationwith a sum of the terms in a shorter window causes bias by modifying the surrogate loss even before backpropagation.

Original: If the windows are sampled i.i.d.
Revised: If the windows are sampled i.i.d.

Original: during the agent’s lifetime, the bias from this effect changes monotonically with the sampling correction coefﬁcient λ , although it may increase or decrease depending on the problem setting if the derivatives are also truncated.
Revised: during the agent’s lifetime, the bias from this effect changes monotonically with the sampling correction coefﬁcient λ , although it may increase or decrease depending on the problem setting if the derivatives are also truncated.

Original: If the i.i.d.
Revised: If the i.i.d.

Original: assumption is violated, the bias could change non-monotonically with λ .
Revised: assumption is violated, the bias could change non-monotonically with λ .

Original: We investigate the effect of sampling correction on the truncated estimators empirically in Section 7.[Figure] [Figure]
Revised: We investigate the effect of sampling correction on the truncated estimators empirically in Section 7.

Original: The sampling correction coefﬁcient λ was introduced to trade off bias and variance, but it is only one possible scheme.
Revised: The sampling correction coefﬁcient λ was introduced to trade off bias and variance, but it is only one possible scheme.

Original: We consider an alternative weighting using exponential discounting motivated  by analogy to the usual discounting procedure in RL.
Revised: We consider an alternative weighting using exponential discounting motivated [Figure] [Figure] by analogy to the usual discounting procedure in RL.

Original: The exponentially discounted meta-gradient estimator is expressed as follows
Revised: The exponentially discounted meta-gradient estimator is expressed as follows

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original:  where α is a meta-discount factor.
Revised: 16) where α is a meta-discount factor.

Original: Analogously to discounting in RL, the weights on the sampling correction terms become exponentially smaller the further back along the update trajectory they are from the return at k .
Revised: Analogously to discounting in RL, the weights on the sampling correction terms become exponentially smaller the further back along the update trajectory they are from the return at k .

Original: Unlike in standard RL, the discount on the sampling correction term uniformly weights all of the terms in a batch, because all the data in the batch arrives to the update Ψ( η, θ, D ) simultaneously.
Revised: Unlike in standard RL, the discount on the sampling correction term uniformly weights all of the terms in a batch, because all the data in the batch arrives to the update Ψ( η, θ, D ) simultaneously.

Original: We will compare this variance reduction scheme with the uniform weighting in
Revised: We will compare this variance reduction scheme with the uniform weighting in

Original: Section 7.
Revised: Section 7.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We conduct experiments in a bandit setting to investigate the bias and variance of the different estimators and how they affect meta-learning performance.
Revised: We conduct experiments in a bandit setting to investigate the bias and variance of the different estimators and how they affect meta-learning performance.

Original: In the experiments, agents are trained with REINFORCE on bandits with randomly sampled arm rewards.
Revised: In the experiments, agents are trained with REINFORCE on bandits with randomly sampled arm rewards.

Original: Each agent collects 30 batches of experience during its lifetime and is updated on each batch except the last one, which is used only to compute the meta-gradient.
Revised: Each agent collects 30 batches of experience during its lifetime and is updated on each batch except the last one, which is used only to compute the meta-gradient.

Original: The meta-learning problem is to set a learning rate parameter for the ﬁrst eight updates and another for the remaining updates such that the total reward achieved by the agent during its lifetime is maximized.
Revised: The meta-learning problem is to set a learning rate parameter for the ﬁrst eight updates and another for the remaining updates such that the total reward achieved by the agent during its lifetime is maximized.

Original: The meta-parameters are optimized over multiple parallel lifetimes and updated after the lifetimes have concluded.
Revised: The meta-parameters are optimized over multiple parallel lifetimes and updated after the lifetimes have concluded.

Original: After the lifetimes have concluded, the agent is reset and new lifetimes are sampled.
Revised: After the lifetimes have concluded, the agent is reset and new lifetimes are sampled.

Original: Intuitively, the problem is to meta-learn an optimal trade-off of exploration and exploitation for the given distribution of bandit tasks.
Revised: Intuitively, the problem is to meta-learn an optimal trade-off of exploration and exploitation for the given distribution of bandit tasks.

Original: Implementation details and hyperparameters can be found in the appendix A.
Revised: Implementation details and hyperparameters can be found in the appendix B.

Original: 7. S AMPLING CORRECTION AND D I CE IN PRACTICE
Revised: 7. S AMPLING CORRECTION AND D I CE IN PRACTICE

Original: In this experiment, we investigate how the different meta-gradient estimators perform in an untruncated setting.
Revised: In this experiment, we investigate how the different meta-gradient estimators perform in an untruncated setting.

Original: In ﬁgure 1, we compare ﬁve different meta-gradient estimators in the bandit setting by training the learning rate parameters starting from nine initializations.
Revised: In ﬁgure 1, we compare ﬁve different meta-gradient estimators in the bandit setting by training the learning rate parameters starting from nine initializations.

Original: Meta-learning curves in the same setting are shown in ﬁgure 2.
Revised: Meta-learning curves in the same setting are shown in ﬁgure 2.

Original: In the meta-learning curves, all algorithms start from the same initialization.
Revised: In the meta-learning curves, all algorithms start from the same initialization.

Original: The ﬁnite differences gradient estimator converges to the local optimum.
Revised: The ﬁnite differences gradient estimator converges to the local optimum.

Original: The naive estimator does not use the sampling correction, which leads to bias, causing the meta-parameters move away from the local optimum.
Revised: The naive estimator does not use the sampling correction, which leads to bias, causing the meta-parameters move away from the local optimum.

Original: The sampling corrected gradient estimator is unbiased, and converges to the same local optimum as the ﬁnite differences estimator.
Revised: The sampling corrected gradient estimator is unbiased, and converges to the same local optimum as the ﬁnite differences estimator.

Original: The gradients of the sampling corrected estimator are similar enough to the ﬁnite differences gradient in this setting that their respective learning curves are almost identical.
Revised: The gradients of the sampling corrected estimator are similar enough to the ﬁnite differences gradient in this setting that their respective learning curves are almost identical.

Original: We use DiCE (Foerster et al., 2018) to represent the category of meta-gradient estimators which use an estimate of the expected Hessian in the metagradient.
Revised: We use DiCE (Foerster et al., 2018) to represent the category of meta-gradient estimators which use an estimate of the expected Hessian in the metagradient.

Original: Since the bandit problem does not have a time dimension, some of the methods developed for reducing variance of the expected Hessian estimator, including the low variance curvature estimator (Rothfuss et al., 2018) and Loaded DiCE (Farquhar et al., 2019), simply reduce to DiCE.
Revised: Since the bandit problem does not have a time dimension, some of the methods developed for reducing variance of the expected Hessian estimator, including the low variance curvature estimator (Rothfuss et al., 2018) and Loaded DiCE (Farquhar et al., 2019), simply reduce to DiCE.

Original: Using DiCE does not stop the meta-optimization from escaping the local optimum, demonstrating that the DiCE meta-gradient is not unbiased by itself.
Revised: Using DiCE does not stop the meta-optimization from escaping the local optimum, demonstrating that the DiCE meta-gradient is not unbiased by itself.

Original: Using DiCE with the sampling corrected meta-gradient estimator converges to a worse local optimum than without DiCE, demonstrating that using the expected Hessian estimator still leads to bias.
Revised: Using DiCE with the sampling corrected meta-gradient estimator converges to a worse local optimum than without DiCE, demonstrating that using the expected Hessian estimator still leads to bias.

Original: Additionally, gradient estimation using
Revised: Additionally, gradient estimation using

Original: DiCE suffers from high variance even in the absence of a time dimension.
Revised: DiCE suffers from high variance even in the absence of a time dimension.

Original: 7. B IAS - VARIANCE TRADEOFF OF THE TRUNCATED ESTIMATORS
Revised: 7. B IAS - VARIANCE TRADEOFF OF THE TRUNCATED ESTIMATORS

Original: We investigate the bias-variance tradeoffs possible for truncated sampling corrected estimators.
Revised: We investigate the bias-variance tradeoffs possible for truncated sampling corrected estimators.

Original: In this experiment, the truncated meta-gradient estimators are computed by randomly sampling a window of one or a few updates during the lifetime and restricting the backpropagation along the update trajectory to that window.
Revised: In this experiment, the truncated meta-gradient estimators are computed by randomly sampling a window of one or a few updates during the lifetime and restricting the backpropagation along the update trajectory to that window.

Original: The objective of the agent is the same as before, i.e., the average lifetime return, which is approximated with the average return within the truncation window.
Revised: The objective of the agent is the same as before, i.e., the average lifetime return, which is approximated with the average return within the truncation window.

Original: There are two caveats to this experiment setting as an approximation of the truncated optimization in a long-horizon setting.
Revised: There are two caveats to this experiment setting as an approximation of the truncated optimization in a long-horizon setting.

Original: One is that, due to the strategy of sampling a ﬁxed window length during the lifetime, the number of times each update index appears in the window is not uniform.
Revised: One is that, due to the strategy of sampling a ﬁxed window length during the lifetime, the number of times each update index appears in the window is not uniform.

Original: These edge effects result in bias in the truncated estimators that would be negligible in a longer-horizon setting.
Revised: These edge effects result in bias in the truncated estimators that would be negligible in a longer-horizon setting.

Original: The other is that in a larger-scale long-horizon setting, it may be impractical to sample the truncation windows i.i.d.
Revised: The other is that in a larger-scale long-horizon setting, it may be impractical to sample the truncation windows i.i.d.

Original: without updating η , which could change how the bias behaves.
Revised: without updating η , which could change how the bias behaves.

Original: In ﬁgure 2, we show the bias and variance of estimators with multiple truncation window lengths and sampling correction coefﬁcients.
Revised: In ﬁgure 2, we show the bias and variance of estimators with multiple truncation window lengths and sampling correction coefﬁcients.

Original: The bias is estimated by computing the average Euclidean distance to the true meta-gradient (computed to high precision with ﬁnite differences) at 25 points around the local optimum of the problem shown in ﬁgure 1.
Revised: The bias is estimated by computing the average Euclidean distance to the true meta-gradient (computed to high precision with ﬁnite differences) at 25 points around the local optimum of the problem shown in ﬁgure 1.

Original: The error bars show the standard deviation of the bias estimated with 10k bootstrap samples from the 25 meta-parameter points, to reﬂect how much the bias varies across the meta-optimisation landscape.
Revised: The error bars show the standard deviation of the bias estimated with 10k bootstrap samples from the 25 meta-parameter points, to reﬂect how much the bias varies across the meta-optimisation landscape.

Original: We know the variance increases with both the sampling correction coefﬁcient and truncation length, so we omit the error bars for variance for clarity.
Revised: We know the variance increases with both the sampling correction coefﬁcient and truncation length, so we omit the error bars for variance for clarity.

Original: Because the sampling correction coefﬁcient and truncation window length change the metagradient magnitude, while its direction may be more important for meta-optimisation in practice, we also present the same data in ﬁgure 5 using cosine similarity as the bias measurement.
Revised: Because the sampling correction coefﬁcient and truncation window length change the metagradient magnitude, while its direction may be more important for meta-optimisation in practice, we also present the same data in ﬁgure 5 using cosine similarity as the bias measurement.

Original: Effect of truncated backpropagation.
Revised: Effect of truncated backpropagation.

Original: Figure 2 shows that the general trend is for the bias to decrease with increased truncation length.
Revised: Figure 2 shows that the general trend is for the bias to decrease with increased truncation length.

Original: However, for the zero sampling correction coefﬁcient the standard deviation of the bias overlaps the mean between multiple truncation lengths suggesting that increasing the truncation length may sometimes increase bias.
Revised: However, for the zero sampling correction coefﬁcient the standard deviation of the bias overlaps the mean between multiple truncation lengths suggesting that increasing the truncation length may sometimes increase bias.

Original: [Figure]
Revised: [Figure]

Original: Effect of sampling correction with the truncated estimator.
Revised: Effect of sampling correction with the truncated estimator.

Original: The monotonic change in the bias due to the sampling correction coefﬁcient discussed in Section 6 is conﬁrmed by the ordering of the mean bias for each truncation length, i.e., even if the bias increases for a subset of the sampling points and decreases for another subset, the mean should always be either increasing or decreasing with the coefﬁcient for a monotonic bias.
Revised: The monotonic change in the bias due to the sampling correction coefﬁcient discussed in Section 6 is conﬁrmed by the ordering of the mean bias for each truncation length, i.e., even if the bias increases for a subset of the sampling points and decreases for another subset, the mean should always be either increasing or decreasing with the coefﬁcient for a monotonic bias.

Original: We ﬁnd that for some of the points in the set of 25 metaparameter values, the bias grows with the sampling correction coefﬁcient.
Revised: We ﬁnd that for some of the points in the set of 25 metaparameter values, the bias grows with the sampling correction coefﬁcient.

Original: The bias getting larger or smaller depending on the sampling point is reﬂected in the large standard deviation of the bias of the truncated estimators.
Revised: The bias getting larger or smaller depending on the sampling point is reﬂected in the large standard deviation of the bias of the truncated estimators.

Original: Variance of the estimators.
Revised: Variance of the estimators.

Original: The variance of the meta-gradient estimators grow rapidly with the sampling correction coefﬁcient.
Revised: The variance of the meta-gradient estimators grow rapidly with the sampling correction coefﬁcient.

Original: Especially the variance of the unbiased estimator is multiple orders of magnitude higher than that of the naive estimator.
Revised: Especially the variance of the unbiased estimator is multiple orders of magnitude higher than that of the naive estimator.

Original: Part of this is explained by the naive estimator having an order of magnitude smaller norm than the true gradient.
Revised: Part of this is explained by the naive estimator having an order of magnitude smaller norm than the true gradient.

Original: In ﬁgure 5, we show that the exponential discounting of the sampling correction terms results in a similar bias-variance tradeoff to the simpler sampling correction coefﬁcient.
Revised: In ﬁgure 5, we show that the exponential discounting of the sampling correction terms results in a similar bias-variance tradeoff to the simpler sampling correction coefﬁcient.

Original: Therefore, at least in this bandit setting, exponential discounting does not seem to have an advantage over using the coefﬁcient.
Revised: Therefore, at least in this bandit setting, exponential discounting does not seem to have an advantage over using the coefﬁcient.

Original: [Equation]
Revised: [Equation]

Original: LEARNING USING SAMPLING CORRECTED TRUNCATED ESTIMATORS
Revised: LEARNING USING SAMPLING CORRECTED TRUNCATED ESTIMATORS

Original: In the previous experiment, we found that the bias of the truncated meta-gradient estimator may increase with the sampling correction coefﬁcient under some conditions.
Revised: In the previous experiment, we found that the bias of the truncated meta-gradient estimator may increase with the sampling correction coefﬁcient under some conditions.

Original: To investigate how using the sampling correction with the truncated estimator works in practice, we train the meta-parameters of the bandits with the different estimators.
Revised: To investigate how using the sampling correction with the truncated estimator works in practice, we train the meta-parameters of the bandits with the different estimators.

Original: We train the meta-parameters for 200k steps using the
Revised: We train the meta-parameters for 200k steps using the

Original: Adam optimizer (Kingma & Ba, 2014).
Revised: Adam optimizer (Kingma & Ba, 2014).

Original: The meta-learning curves shown in ﬁgure 3 suggest that the higher sampling correction values asymptote to better local optima.
Revised: The meta-learning curves shown in ﬁgure 3 suggest that the higher sampling correction values asymptote to better local optima.

Original: Therefore, even though the sampling correction may sometimes increase bias of the truncated estimators it can still lead to better meta-learning performance in practice.
Revised: Therefore, even though the sampling correction may sometimes increase bias of the truncated estimators it can still lead to better meta-learning performance in practice.

Original: We conﬁrm that using Adam does not change the ordering of the results by repeating the experiment with SGD, with results shown in ﬁgure 4.
Revised: We conﬁrm that using Adam does not change the ordering of the results by repeating the experiment with SGD, with results shown in ﬁgure 4.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we investigated the estimation of meta-gradients in meta-RL.
Revised: In this paper, we investigated the estimation of meta-gradients in meta-RL.

Original: We showed that, unlike claims in prior work, using an estimator of the expected Hessian in the meta-gradient estimator always adds bias, and is likely to also increase variance.
Revised: We showed that, unlike claims in prior work, using an estimator of the expected Hessian in the meta-gradient estimator always adds bias, and is likely to also increase variance.

Original: Further, we discussed the sampling correction needed for unbiased meta-gradient estimates, and described how it may be employed in the truncated optimization setting.
Revised: Further, we discussed the sampling correction needed for unbiased meta-gradient estimates, and described how it may be employed in the truncated optimization setting.

Original: In doing so, we explored the space of bias-variance tradeoffs that can be made in meta-gradient estimation.
Revised: In doing so, we explored the space of bias-variance tradeoffs that can be made in meta-gradient estimation.

Original: We found that applying the sampling correction to truncated meta-gradients leads to convergence to better local optima in our setting, although it increases variance and does not decrease the bias of the truncated gradient estimator in all cases.
Revised: We found that applying the sampling correction to truncated meta-gradients leads to convergence to better local optima in our setting, although it increases variance and does not decrease the bias of the truncated gradient estimator in all cases.

Original: While these ﬁndings can guide researchers to optimize the bias-variance tradeoff for their setting, work remains to be done in extending the investigation to the full RL setting with a time dimension in the inner loop, and developing better meta-gradient estimators that reduce both the bias and variance of the naive estimator. 
Revised: While these ﬁndings can guide researchers to optimize the bias-variance tradeoff for their setting, work remains to be done in extending the investigation to the full RL setting with a time dimension in the inner loop, and developing better meta-gradient estimators that reduce both the bias and variance of the naive estimator. A CKNOWLEDGMENTS AND DISCLOSURE OF FUNDING Risto Vuorio is supported by EPSRC Doctoral Training Partnership Scholarship and Department of Computer Science Scholarship.

Original:  A B ANDIT SETTINGS
Revised: Jakob Beck is supported by the Oxford-Google DeepMind Doctoral Scholarship. This work was supported by a generous equipment grant from NVIDIA A D ERIVATION OF THE UNBIASED META - GRADIENT

Revised: More detailed derivation of the unbiased meta-gradient estimator is given by

Revised: [Equation]

Revised: sampling correction

Revised: [Equation]

Revised: B B ANDIT SETTINGS

Original: In this section we describe the details of the bandit experiments.
Revised: In this section we describe the details of the bandit experiments.

Original: The experiments are conducted in a multi-armed bandit setting.
Revised: The experiments are conducted in a multi-armed bandit setting.

Original: The bandits have 30 arms with the arm means sampled from exp ( Uniform ( − 100 , 1)) .
Revised: The bandits have 30 arms with the arm means sampled from exp ( Uniform ( − 100 , 1)) .

Original: The reward for each pull is computed by adding noise sampled from
Revised: The reward for each pull is computed by adding noise sampled from

Original: Gaussian with standard deviation 2 to the arm mean.
Revised: Gaussian with standard deviation 2 to the arm mean.

Original: The policy is parametrized as a softmax over a randomly initialized vector of logits.
Revised: The policy is parametrized as a softmax over a randomly initialized vector of logits.

Original: The inner learning problem is learning the policy for the bandit and the outer problem is to learn a learning rate schedule for the inner learner.
Revised: The inner learning problem is learning the policy for the bandit and the outer problem is to learn a learning rate schedule for the inner learner.

Original: The inner loop uses a simple policy gradient algorithm (Williams, 1992).
Revised: The inner loop uses a simple policy gradient algorithm (Williams, 1992).

Original: The inner loop updates the policy for 29 update steps sampling 30 batches of experience in total.
Revised: The inner loop updates the policy for 29 update steps sampling 30 batches of experience in total.

Original: Each update in the inner loop is computed on ten samples from the bandit.
Revised: Each update in the inner loop is computed on ten samples from the bandit.

Original: The outer loop updates the learning rate schedule with the meta-gradient computed by sampling from an expression similar to 10.
Revised: The outer loop updates the learning rate schedule with the meta-gradient computed by sampling from an expression similar to 8.

Original: The outer loop objective used in practice considers the return in each batch sampled from the bandit instead of the last one alone.
Revised: The outer loop objective used in practice considers the return in each batch sampled from the bandit instead of the last one alone.

Original: When truncation is applied, the inner loop is run as before but the outer loop only considers returns within the truncation window and only backpropagates gradients within the window.
Revised: When truncation is applied, the inner loop is run as before but the outer loop only considers returns within the truncation window and only backpropagates gradients within the window.

Original: The meta-parameter is a two-dimensional array, from which the learning rate is computed by choosing one of the two values according to the index of the update on the inner loop lifetime and applying the softplus function.
Revised: The meta-parameter is a two-dimensional array, from which the learning rate is computed by choosing one of the two values according to the index of the update on the inner loop lifetime and applying the softplus function.

Original: The ﬁrst 8 updates correspond to the ﬁrst dimension of the meta-parameter and the rest to the second.
Revised: The ﬁrst 8 updates correspond to the ﬁrst dimension of the meta-parameter and the rest to the second.

Original: The meta-gradient is computed across multiple inner learning problems in parallel.
Revised: The meta-gradient is computed across multiple inner learning problems in parallel.

Original: For computing one meta-gradient update with the untruncated inner loop, inner batch size × lifetime length × parallel runs samples from the bandit are used.
Revised: For computing one meta-gradient update with the untruncated inner loop, inner batch size × lifetime length × parallel runs samples from the bandit are used.

Original: All the updates are computed with stochastic gradient descent.
Revised: All the updates are computed with stochastic gradient descent.

Original: The hyperparameters of the experiments in each ﬁgure are summarized in 1.
Revised: The hyperparameters of the experiments in each ﬁgure are summarized in 1.

Original: B A DDITIONAL EXPERIMENTAL RESULTS
Revised: C A DDITIONAL EXPERIMENTAL RESULTS

Original: The meta-gradient estimators are compared in terms of cosine similarity in the left panel of ﬁgure 5.
Revised: The meta-gradient estimators are compared in terms of cosine similarity in the left panel of ﬁgure 5.

Original: The two weighting schemes for the sampling correction are compared in the right panel of ﬁgure 5.
Revised: The two weighting schemes for the sampling correction are compared in the right panel of ﬁgure 5.

Original: Learning curves for the truncated meta-gradient estimators using SGD as the optimizer are shown in 
Revised: Learning curves for the truncated meta-gradient estimators using SGD as the optimizer are shown in ﬁgure 4.

Revised: The large error in the curves for the higher truncation lengths are due to some of the seeds

Original: [Figure]
Revised: [Figure]

Original: Truncation length

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Truncation length

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ﬁgure 4.
Revised: [Figure]

Original: The large error in the curves for the higher truncation lengths are due to some of the seeds becoming unstable during training and deviating far from the optimal region in a single gradient step.
Revised: [Figure] [Figure] becoming unstable during training and deviating far from the optimal region in a single gradient step.

Original: This effect can be reduced in practice by using adaptive optimizers such as Adam and gradient clipping.
Revised: This effect can be reduced in practice by using adaptive optimizers such as Adam and gradient clipping.

Revised: [Figure]