A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learning
Abstract
Gradient-based Meta-RL (GMRL) refers to methods that maintain two-level optimisation procedures wherein the outer-loop meta-learner guides the inner-loopgradient-based reinforcement learner to achieve fast adaptations.
In this paper, wedevelop a unified framework that describes variations of GMRL algorithms andpoints out that existing stochastic meta-gradient estimators adopted by GMRL areactually biased .
Such meta-gradient bias comes from two sources: 1) the compositional bias incurred by the two-level problem structure, which has an upper boundof O (cid:0)
𝐾𝛼 𝐾 ˆ 𝜎
In | 𝜏 | − 0 .
5 (cid:1) w.r.t.
inner-loop update step 𝐾 , learning rate 𝛼 , estimatevariance ˆ 𝜎
In and sample size | 𝜏 | , and 2) the multi-step Hessian estimation bias
ˆ Δ 𝐻due to the use of autodiff, which has a polynomial impact O (cid:0) ( 𝐾 − 1 )( ˆ Δ 𝐻 )
𝐾 − 1 (cid:1) onthe meta-gradient bias.
We study tabular MDPs empirically and offer quantitativeevidence that testifies our theoretical findings on existing stochastic meta-gradientestimators.
Furthermore, we conduct experiments on Iterated Prisoner’s Dilemmaand Atari games to show how other methods such as off-policy learning andlow-bias estimator can help fix the gradient bias for GMRL algorithms in general.
Introduction
Meta Learning, also known as learning to learn, is proposed to equip intelligent agents with metaknowledge for fast adaptations [29].
Meta-Reinforcement Learning (RL) algorithms aim to train RLagents that can adapt to new tasks with only few examples [5, 10, 16].
For example, MAML-RL[10], a typical Meta-RL algorithm, learns the initial parameters of an agent’s policy so that the agentcan rapidly adapt to new environments with a limited number of policy-gradient updates.
Recently,
Meta-RL algorithms have been further developed beyond the scope of learning fast adaptations.
Animportant direction is to conduct online meta-gradient learning for adaptively tuning algorithmichyper-parameters [38, 40] or designing intrinsic reward [41] during training in one single task.
Besides, there are Meta-RL developments that manage to discover new RL algorithms by learningalgorithmic components or other fundamental concepts in RL, such as the policy gradient objective[26] or the TD-target [39], which can generalise to solve a distribution of different tasks.
In general, gradient-based Meta-RL (GMRL) tasks can be formulated by a two-level optimisationprocedure.
This procedure optimises the parameters of an outer-loop meta-learner, whose objectiveis dependent on a 𝐾 -step policy update process (e.g., stochastic gradient descent) conducted by aninner-loop learner.
Formally, this procedure can be written as:
[Equation]
where 𝜽 are inner-loop policy parameters, 𝝓 are meta parameters, 𝛼 is the learning rate, 𝐽
In and𝐽 Out are value functions for the inner and the outer-loop learner.
In solving Eq.
(1), estimatingthe meta-gradient of ∇ 𝝓 𝐽 𝐾 ( 𝝓 ) from a two-level optimisation process is non-trivial; how to conductaccurate estimation on ∇ 𝝓 𝐽 𝐾 ( 𝝓 ) has been a critical yet challenging research problem [23, 28, 32].
In this paper, we point out that the meta-gradient estimators adopted by many recent GMRL methods[26, 39] are in fact biased.
We conclude that such bias comes from two sources: (1) the compositionalbias and (2) the multi-step Hessian bias .
The compositional bias origins from the discrepancybetween the sampled policy gradient ∇ 𝜽 ˆ 𝐽
In and expected policy gradient ∇ 𝜽 𝐽
In , and the multistep Hessian estimation bias occurs due to the biased Hessian estimation ∇ 2 𝜽 ˆ 𝐽
In resulting from theemployment of automatic differentiation in modern GMRL implementations.
There are very few prior work that investigates the bias in GMRL, most of them limited to the Hessianestimation bias in the MAML-RL setting [28, 32].
Our paper investigates the above two biases in abroader setting and applies on generic meta-gradient estimators.
For the compositional bias term,we offer the first theoretical analysis on its quantity, based on which we then investigate currentmitigation solutions.
For the multi-step Hessian bias, we provide rigorous analysis in GMRL settingsparticularly for those GMRL tasks where complex inner-loop optimisations are involved.
For the rest of the paper, we first introduce a unified Meta-RL framework that can describe variationsof existing GMRL algorithms.
Building on this framework, we offer two theoretical results that 1) thecompositional bias has an upper bound of O (cid:0)
𝐾𝛼 𝐾 ˆ 𝜎
In | 𝜏 | − 0 .
5 (cid:1) with respect to the inner-loop updatestep 𝐾 , the learning rate 𝛼 , the estimate variance ˆ 𝜎
In and the sample size | 𝜏 | , and 2) the multi-step
Hessian bias ˆ Δ 𝐻 has a polynomial impact of O (cid:0) ( 𝐾 − 1 )( ˆ Δ 𝐻 ) 𝐾 − 1 (cid:1) .
To validate our theoretical insightson these two biases, we conduct a comprehensive list ablation studies.
Experiments over tabular MDPwith MAML-RL [10] and LIRPG [41] demonstrate how quantitatively these two biases influence theestimation accuracy, which consolidates our theories.
Furthermore, our theoretical results help understand to what extent existing methods can mitigatetheses two bias empirically.
For the compositional bias, we show that off-policy learning methods canreduce the inner-loop policy gradient variance and the resulting compositional bias.
For the multi-step
Hessian bias, we study how the low-variance curvature technique based on Rothfuss et al.
[28] canhelp correct the Hessian bias for general GMRL problems.
We test these solutions on environmentsincluding iterated Prisoner’s Dilemma with off-policy corrected LOLA-DiCE [13], and eight Atarigames based on MGRL [38] with the multi-step Hessian correction technique.
Experimental resultsconfirm that those bias-correction methods can substantially decrease the meta-gradients bias andimprove the overall performance on rewards.
Related Work
A pioneering work that studies meta-gradient estimation is Al-Shedivat et al.
[1] who discussed thebiased estimation problem of MAML and proposed the E-MAML formulation to fix the meta-gradientbias.
Following work includes Rothfuss et al.
[28], Liu et al.
[23], Tang et al.
[32] that tried to fix themeta-gradient estimation error by reducing the estimation variance so as to improve the performance.
Moreover, Foerster et al.
[12], Farquhar et al.
[8], Mao et al.
[24] discussed the higher-order gradientestimation in RL.
Recently, Bonnet et al.
[4], Vuorio et al.
[35] proposed algorithms to balancethe bias-variance trade-off for meta-gradient estimates.
Our work is different from all the aboveprior work from three folds: (1) we study the additional bias term (the compositional bias) (2) weconsider different formulation (expected update while sample-based update in [35], refer to Appendix
B for more discussions) (3) we study a broader scope of applications that include different GMRLalgorithm instantiations and settings.
In the following part, we review existing GMRL algorithms and categorise them into four researchtopics.
We offer one typical example for each topic in Table 1 and further discussed their relationshipand how they can be unified into one framework in Sec. 3.
We also provide a more self-containedexplanation for each topic in Appendix A for readers that are not familiar with GMRL.
Few-shot RL.
The idea of few-shot RL is to enable RL agent with fast learning ability.
Specifically,the RL agent is only allowed to interact with the environment for a few trajectory to conduct taskspecific adaptation.
Conducting few-shot RL has two approaches: gradient based and context based.
Context based Meta-RL involves works [5, 15, 27, 36], which uses neural network to embed the information from few-shot interactions so as to obtain task-relevant context.
Gradient based few-shot
RL [1, 23, 28] focus on meta-learning the model’s initial parameters through meta-gradient descent.
Opponent Shaping.
Foerster et al.
[13], Letcher et al.
[22], Kim et al.
[19] explicitly models thelearning process of opponents in multi-agent learning problems, which can be thought of as GMRLproblems since meta-gradient estimation in these works involve differentiation over opponent’s policyupdates.
By modelling opponents’ learning process, the multi-agent learning process can reach bettersocial welfare [13, 22] or the ego agent can adapt to a new peer agent [19].
Single-lifetime Meta-gradient RL.
This line of research focuses on learning online adaptationover algorithmic hyper-parameters to enhance the performance of an RL agent in one single task,such as discount factor [38, 40], intrinsic reward generator [41], auxiliary loss [33], reward shapingmechanism [17], and value correction [43].
The main feature of online meta-gradient RL is that theyare under single-lifetime setting [39], meaning that the algorithm only iterates through the whole RLlearning procedure in one task rather a distribution of tasks.
Multi-lifetime Meta-gradient RL.
Compared to the previous single-lifetime settings, “multilifetime" refers to settings where agents learn to adapt on a distribution of tasks or environments.
Themeta-learning target includes policy gradient or TD learning objectives [3, 20, 26], intrinsic reward[42], target value function [39], options in hierarchical RL [34], and recently the design of curriculumin multi-agent learning [9].
A Unified Framework for Meta-gradient Estimation
In this section, we derive a general formulation for meta-gradient estimation in GMRL.
This formulation enables us to conduct general analysis about meta-gradient estimation and we will show howalgorithms in the four research topics mentioned in Sec.
2 can be described through it.
We propose that a general GMRL objective with 𝐾 -step inner-loop policy gradient update can bewritten as the following objective
[Equation]
We denote the meta-parameters as 𝝓 , and the pre- and post-adapt inner parameters as 𝜽 and 𝜽 𝐾 ,respectively.
The meta objective as 𝐽 Out , the inner loop objective as 𝐽 In .
The general objective for
GMRL is to maximise the meta objective 𝐽 Out ( 𝝓 , 𝜽 𝐾 ) , where the inner-loop post-adapt parametersare obtained by taking 𝐾 policy gradient steps.
Note that all inner-loop updates refer to an expectedpolicy gradient (EPG) and all bias term we discuss hereafter is the bias w.r.t the exact meta-gradientin this 𝐾 -step EPG inner-loop setting.
We discuss the truncated 𝐾 -step EPG setting in Appendix B.
The form of exact meta-gradient ∇ 𝝓 𝐽 𝐾 ( 𝝓 ) is given by the following proposition via the chain rule.
Proposition 3.1 ( 𝐾 -step Meta-Gradient) .
The exact meta-gradient to the objective in Eq.
(2) can bewritten as:
[Equation]
[Equation]
The derivation of the above proposition is in Appendix F.1.
For each topic mentioned in Sec. 2, we pick one GMRL algorithm example to illustrate how they canbe fit into this framework.
To describe meta-gradient estimation in RL, we start with basic notations.
Consider a discrete-time finite horizon Markov Decision Process (MDP) defined by ⟨S , A , 𝑝, 𝑟, 𝛾, 𝐻 ⟩ .
At each time step 𝑡 , the RL agent observes a state 𝒔 𝑡 ∈ S , takes an action 𝒂 𝑡 ∈ A based on the policy𝜋 𝜽 ( 𝒂 𝑡 | 𝒔 𝑡 ) parametrised with 𝜽 ∈ ℝ 𝑑 , transits to the next state 𝒔 𝑡 + 1 ∈ S according to the transition function 𝑝 ( 𝒔 𝑡 + 1 | 𝒔 𝑡 , 𝒂 𝑡 ) and receives the reward 𝑟 ( 𝒔 𝑡 , 𝒂 𝑡 ) .
We define the return R( 𝝉 ) =
(cid:205)
𝐻 𝑡 =
𝛾 𝑡 𝑟 ( 𝒔 𝑡 , 𝒂 𝑡 )as the discounted sum of rewards along a trajectory 𝝉 : =
( 𝒔 0 , 𝒂 0 , . . . , 𝒔 𝐻 − 1 , 𝒂 𝐻 − 1 , 𝒔 𝐻 ) sampledby agent policy.
The objective for the RL agent is to maximise the expected discounted sum ofrewards 𝑉 ( 𝜽 ) =
𝔼 𝝉 ∼ 𝑝 ( 𝝉 ; 𝜽 )
[R( 𝝉 )] .
Then the RL agent updates parameter 𝜽 using policy gradientgiven by ∇ 𝜽 𝑉 ( 𝜽 ) =
𝔼 𝝉 ∼ 𝑝 ( 𝝉 ; 𝜽 ) [∇ 𝜽 log 𝜋 𝜽 ( 𝝉 )R( 𝝉 )] where ∇ 𝜽 log 𝜋 𝜽 ( 𝝉 ) =
(cid:205)
𝐻𝑡 = 0 ∇ 𝜽 log 𝜋 𝜽 ( 𝒂 𝑡 | 𝒔 𝑡 ) .
For
[Table]
two-agent RL problems, we can extend the MDP to two-agent MDP (or, Stochastic games [30])defined by ⟨S 1 , S 2 , A 1 , A 2 , 𝑃, 𝑟 1 , 𝑟 2 , 𝛾, 𝐻 ⟩ , the learning objective for agent 𝑖 is to maximise its valuefunction of 𝑉 𝑖 ( 𝜽 ) =
𝔼 𝝉 𝑖 ∼ 𝑝 ( 𝝉 𝑖 ; 𝜽 𝑖 )
[R 𝑖 ( 𝝉 𝑖 )] .
MAML.
Finn et al.
[10] optimized over meta initial parameters to maximize the return of one-stepadapted policy: 𝜽 1 = 𝜽 + 𝛼 ∇ 𝜽 𝑉 ( 𝜽 ) .
In MAML-RL, 𝐽 Out ( 𝝓 , 𝜽 1 ) degenerates to 𝑉 ( 𝜽 1 ) and 𝝓 and𝜽 represent the same initial parameters.
The meta-gradient can be derived in the form of Eq.
(3):∇ 𝜽 𝜽 1 ∇ 𝜽 1 𝑉 ( 𝜽 1 ) , where ∇ 𝜽 𝜽 1 =
𝐼 + 𝛼 ∇ 2 𝜽 𝑉 ( 𝜽 ) .
LOLA.
Foerster et al.
[13] proposed a new learning objective by including an additional term thataccounts for the impact of the ego policy to the anticipated opponent’s gradient update.
For LOLAagent with parameters 𝝓 , it will optimise its return over one-step-lookahead opponent parameters 𝜽 1 .
The meta-gradient can be shown as: ∇ 𝝓 𝑉 1 ( 𝝓 , 𝜽 1 )+∇ 𝝓 𝜽 1 ∇ 𝜽 1 𝑉 1 ( 𝝓 , 𝜽 1 ) , where ∇ 𝝓 𝜽 1 =
𝛼 ∇ 𝝓 ∇ 𝜽 𝑉 2 ( 𝝓 , 𝜽 )
MGRL.
Xu et al.
[38] proposed to tune the discount factor 𝛾 and bootstrapping parameter 𝜆 inan online manner.
The main feature of MGRL is to conduct inner-loop RL policy 𝜽 update andouter-loop meta parameters 𝝓
= ( 𝛾, 𝜆 ) alternately.
In MGRL, 𝐽 Out ( 𝝓 , 𝜽 1 ) degenerates to 𝑉 ( 𝜽 1 ) .
Themeta-gradient takes the form: ∇ 𝝓 𝜽 1 ∇ 𝜽 1 𝑉 ( 𝜽 1 ) , ∇ 𝝓 𝜽 1 = 𝛼 ∇ 𝝓 ∇ 𝜽 𝑉 ( 𝝓 , 𝜽 ) .
LPG.
Oh et al.
[26] aimed to learn a neural network based RL algorithm, by which a RL agentcan be properly trained.
In LPG, 𝜽 represents the RL agent policy parameters and 𝝓 is the metaparameter of neural LSTM RL algorithm,
𝐽
In ( 𝝓 , 𝜽 ) denotes 𝑓 ( 𝝓 , 𝜽 ) , which is the output of metanetwork 𝝓 for conducting inner-loop neural policy gradients.
The meta-gradient can be shown as:
[Equation]
Remark.
The analytical form of exact meta-gradient given in Eq.
(3) involves computation offirst-order gradient ∇ 𝜽 𝐽
In , ∇ 𝝓 𝐽 Out and ∇ 𝜽
𝐽
Out , Jacobian ∇ 𝝓 ∇ 𝜽 𝐽
In and Hessian ∇ 2 𝜽 𝐽
In .
In practice,these four quantities can be estimated by random samples from inner-loop update step 0 to 𝐾 − 1 ,which denoted by 𝝉 0: 𝐾 − 1, 𝝉 0: 𝐾 − 1, 𝝉 0: 𝐾 − 1, 𝝉 3 .
As a result, the estimated gradient ∇ 𝝓 ˆ 𝐽 𝐾 ( 𝝓 ) can bederived as:
[Equation]
The post-adapt inner parameter estimate takes the form ˆ 𝜽 𝐾 = 𝜽 0 + 𝛼 (cid:205)
𝐾 − 1 𝑖 = 0 ∇ ˆ 𝜽 𝑖 ˆ 𝐽
In ( 𝝓 , ˆ 𝜽 𝑖 , 𝝉 𝑖 0 ) .
Theoretical Analysis of Meta-gradient Estimators
In this section, we systematically discuss and theoretically analyse the bias and variance terms formeta-gradient estimations in the current GMRL literature.
We highlight two important sources ofbiases in meta-gradient estimations: the compositional bias and the multi-step Hessian bias.
Ouranalysis builds on the following three assumptions:
Assumption 4.1 (Lipschitz continuity) .
The outer-loop objective function 𝐽 Out satisfies that𝐽 Out (· , 𝜽 ) and 𝐽 Out ( 𝝓 , ·) are Lipschitz continuous with constants 𝑚 𝜽 and 𝑚 𝝓 respectively, 𝑚 1 =max 𝜽 𝑚 𝜽 , 𝑚 2 = max 𝝓 𝑚 𝝓 .
∇ 𝝓 𝐽 Out (· , 𝜽 ) and ∇ 𝜽 𝐽 Out ( 𝝓 , ·) are Lipschitz continuous with constants𝜇 𝜽 and 𝜇 𝝓 respectively, 𝜇 1 = max 𝜽 𝜇 𝜽 , 𝜇 2 = max 𝝓 𝜇 𝝓 .
The inner-loop objective function 𝐽 Insatisties that ∇ 𝜽
𝐽
In (· , 𝜽 ) and ∇ 𝜽 𝐽
In ( 𝝓 , ·) are Lipschitz continuous with constants 𝑐 𝜽 and 𝑐 𝝓 respectively,𝑐 1 = max 𝜽 𝑐 𝜽 , 𝑐 2 = max 𝝓 𝑐 𝝓 .
∇ 2 𝜽 𝐽
In ( 𝝓 , ·) is Lipschitz continuous with constants 𝜌 𝝓 , 𝜌 2 = max
𝝓 𝑚 𝝓 .
Assumption 4.2 (Bias of estimators) .
Outer-loop stochastic gradient estimator ∇ 𝝓 ˆ 𝐽 Out
( 𝝓 , 𝜽 , 𝝉 ) and∇ 𝜽
ˆ 𝐽 Out
( 𝝓 , 𝜽 , 𝝉 ) are unbiased estimator of ∇ 𝝓 𝐽 Out ( 𝝓 , 𝜽 ) and ∇ 𝜽 𝐽 Out ( 𝝓 , 𝜽 ) .
Inner-loop stochasticgradient estimator ∇ 𝜽 ˆ 𝐽
In ( 𝝓 , 𝜽 , 𝝉 ) is unbiased estimator of ∇ 𝜽 𝐽
In ( 𝝓 , 𝜽 ) .
Assumption 4.3 (Variance of estimators) .
The outer-loop stochastic gradient estimator∇ 𝝓 ˆ 𝐽 Out ( 𝝓 , 𝜽 , 𝝉 ) and ∇ 𝜽 ˆ 𝐽 Out
( 𝝓 , 𝜽 , 𝝉 ) has bounded variance, i.e., 𝔼 𝜏 [∥∇ 𝝓 ˆ 𝐽 Out ( 𝝓 , · , 𝝉 )
−∇ 𝝓 𝐽 Out ( 𝝓 , ·)
∥ 2 ] ≤ ( 𝜎 1 ) 2 , and 𝔼 𝜏 [∥∇ 𝜽 ˆ 𝐽 Out ( 𝝓 , · , 𝝉 ) − ∇ 𝜽 𝐽 Out ( 𝝓 , ·)∥ 2 ] ≤ ( 𝜎 2 ) 2 .
The above three assumptions are all common ones adopted by existing work [6, 7, 18].
4.1 The Compositional Bias
Recall the 𝐾 -step inner-loop meta-gradient estimate in Eq.
(4), existing GMRL methods usually getunbiased outer-loop gradient estimator ∇ 𝝓 ˆ 𝐽 Out ( 𝝓 , ˆ 𝜽 𝐾 , 𝝉 3 ) and ∇ ˆ 𝜽 𝐾 ˆ 𝐽 Out
( 𝝓 , ˆ 𝜽 𝐾 , 𝝉 3 ) , then the algorithmcan get unbiased meta-gradient estimation by plugging in unbiased inner-loop gradient estimation ˆ 𝜽 𝐾 ,where 𝔼 [ ˆ 𝜽 𝐾 ] = 𝜽 𝐾 .
However, this is not true because of the compositional optimisation structure.
Consider a non-linear compositional scalar objective 𝑓 ( 𝜽 𝐾 ) , the gradient estimation bias comes fromthe fact that
[Equation]
If one substitutes the non-linear function 𝑓 ( 𝜽 𝐾 ) with ∇ 𝜽 𝐾 𝐽 Out ( 𝝓 , 𝜽 𝐾 ) and ∇ 𝝓 𝐽 Out ( 𝝓 , 𝜽 𝐾 ) , then atypical meta-gradient estimation in GMRL introduces compositional bias:
[Equation]
which leads to meta-gradient estimation bias.
The following lemma characterises compositional bias.183 Lemma 4.4 (Compositional Bias) .
Suppose that Assumption 4.1 and 4.2 hold, let ˆ Δ 𝐶 =
𝔼 [∥ 𝑓 ( ˆ 𝜽 {+𝐾 ) −𝑓 ( 𝜽 𝐾 )∥] be the compositional bias and 𝐶 
+}
the Lipschitz constant of 𝑓 (·) , | 𝝉 | denote number oftrajectories used to estimate inner-loop gradient in each inner-loop update step, 𝛼 the learning rate,then we have,
[Equation]
where ˆ 𝜎 In = max 𝑖 √︃ 𝕍 [∇ 𝜽 𝑖 ˆ 𝐽
In ( 𝝓 , 𝜽 𝑖 , 𝝉 𝑖 0 )] , 𝑖 ∈ { 0 , .. [Equation] 𝐾 − 1 } .
Proof.
See Appendix G.1 for a detailed proof.
□
Lemma 4.4 indicates that the compositional bias comes from the inner-loop policy gradient estimate,concerning the learning rate 𝛼 , the sample size | 𝝉 | and the variance of policy gradient estimator ˆ 𝜎
In .
This is a fundamental issue in many existing GMRL algorithms [13, 38] since applying stochasticpolicy gradient update can introduce estimation errors, possibly due to large sampling variance,therefore ˆ 𝜽 𝐾 ≠ 𝜽 𝐾 .
It also implies that the bias issue becomes more serious under the multi-stepformulation since each policy gradient step introduces estimation error, resulting in composite biases.
4.2 The Multi-step Hessian Bias
Recall the analytical form of the exact meta-gradient in Eq.
(3), estimating ∇ 𝝓 𝜽 𝐾 involves computing Hessian ∇ 2 𝜽 𝑗 𝐽
In ( 𝝓 , 𝜽 𝑗 ) .
In Eq.
(4), the Hessian term is estimated by ∇ 2ˆ 𝜽 𝑗 𝐽
In ( 𝝓 , ˆ 𝜽 𝑗 , 𝜏 𝑗 2 ) where 𝑗 ∈{ 1 , . . .
, 𝐾 − 1 } .
Hessian estimation is a non-trivial problem in GMRL, biased Hessian estimationissue has been brought up in various MAML-RL papers[23, 28, 32], we offer a brief summary in
Appendix C due to page limit.
Beyond MAML-RL, many recent GMRL work suffers from the samebias due to applying direct automatic differentiation.
For example, existing work such as Eq.
(3) in Oh et al.
[26], Eq.
(4) in Zheng et al. [42], Eq.
(12), (13), (14) in Xu et al.
[38] and Eq.
(5), (6), (7) in Xu et al.
[39] suffers from this issue.
Interestingly, most of them are coincidentally unbiased if theyonly conduct only one-step policy gradient update in the inner-loop.
For 𝐾 -step GMRL when 𝐾
= 1 ,∇ 𝝓 𝜽 1 in meta-gradient writes as:
[Equation]
We can see from Eq.
(7) that it would not involve Hessian ∇ 2 𝐽 In computation if 𝝓 ≠ 𝜽 .
To furtherillustrate, in one-step MGRL, we can show that the estimation of ∇ 𝝓 𝜽 1 are unbiased because it takesderivatives w.r.t meta-parameters 𝝓 = ( 𝛾, 𝜆 ) which don’t have gradient dependency on the trajectorydistribution.
However, when it takes more than one-step inner-loop policy gradient updates, themeta-gradient estimation will get the hessian estimation bias.
As a result, for the reason that when𝐾 > 1 in 𝐾 -step GMRL, the ∇ 𝝓 𝜽 𝐾 in meta-gradient takes the form:
[Equation]
This is the reason why we name it by multi-step Hessian bias.
4.3 Theoretical Bias-Variance Analysis
Based on Lemma 4.4 and the discussion in Sec.
4.2, we can derive the upper bound on the bias andvariance of the meta-gradient with 𝐾 -step inner-loop updates.
Theorem 4.5 (Upper bound for the bias and the variance) .
Suppose that Assumption 4.1 and 4.2and 4.3 hold.
Let 𝐽 𝝓 , 𝜽 denote ∇
𝝓 ∇ 𝜽 𝐽
In , 𝐻 𝜽 , 𝜽 denote ∇ 2 𝜽 𝐽
In , ˆ Δ 𝐾 = ∥
𝔼 [∇ 𝝓 ˆ 𝐽 𝐾 ( 𝝓 )] − ∇ 𝝓 𝐽 𝐾 ( 𝝓 )∥ be themeta-gradient estimation bias, set 𝐵
=
+ 𝛼𝑐 2 .
Then the bound of bias hold:
[Equation]
[Equation]
Let ( ˆ 𝜎 𝐾 ) [Equation] =
𝕍[-(cid:2) ∇ 𝝓 ˆ 𝐽 𝐾 ( 𝝓 ) (cid:3) be the meta-gradient estimation variance, set 𝑉 
 = ( 1 + 𝛼𝑐 2 ) 2 , 𝑉 2 =2 𝛼 2 ( 𝑚 21 + 3 𝜎 22 ) the estimation variance is given by-]
[Equation]
∇ 𝝓 ˆ 𝐽 𝐾 ( 𝝓 {+) (cid:3) be the meta-gradient estimation variance, set 𝑉 
+} =
( 1 + 𝛼𝑐 2 ) 2 , 𝑉 2 =2 𝛼 2 ( 𝑚 21 + 3 𝜎 22 ) the estimation variance is given by
[Equation]
where ˆ
[Equation]
𝐽
[Equation]
[Equation]
ˆ 𝜎 𝐻
[Equation]
max 𝜽 𝕍 [ ˆ 𝐻 𝜽
[Equation]
| 𝜏 | .
Proof.
See Appendix H.1 for a detailed proof.
□
Theorem 4.5 shows that the upper bound of bias and variance consists of two parts: the first termindicates compositional bias 𝔼 [∥ ˆ 𝜽 𝐾 − 𝜽 𝐾 ∥] in Lemma 4.4, while the second term refers to thesecond-order estimation bias ( ˆ Δ 𝐽 , ˆ Δ 𝐻 ) and the variance ( ˆ 𝜎 𝐽 , ˆ 𝜎 𝐻 ).
Several observations can be madebased on Theorem 4.5: 1) the compositional bias exists in both upper bound of bias and variance;2) the multi-step Hessian bias has polynomial impact on the upper bound of bias and variance; 3)most importantly, many existing GMRL algorithms suffer from the compositional bias; moreover, the
Hessian bias can significantly increase meta-gradient bias in the multi-step inner-loop setting.
4.4 Understanding Existing Mitigations for Meta-gradient Biases
Based on the above theoretical analysis, we now try to explore and understand how existing methodscan handle these two estimation biases.
Off-policy Learning.
From the Lemma 4.4 and the discussion in Sec. 4.1, we know that thecompositional bias is caused by the estimation error between ˆ 𝜃 𝐾𝑡 and 𝜃 𝐾𝑡 .
In this case, a simple idea isthat one can leverage off-policy learning technique [31] to handle the compositional bias problem byreusing samples 𝜏 𝑖 0: 𝑡 − 1 , 0 together with 𝜏 𝑖𝑡, 0 to approximate 𝜃 𝐾𝑡 , we want ˆ 𝜃 𝐾𝑡 to stay close to 𝜃 𝐾𝑡 .
Theintuition behind this correction is to enlarge the sample size | 𝝉 | so that the compositional bias can beminimised according to Lemma 4.4.
Specifically, we can apply importance sampling technique tocorrect the compositional bias, i.e., 𝔼 𝝉 ∼ 𝑝 ( 𝝉 ; 𝜽 ) (cid:104)(cid:205)
𝐻 − 1 𝑡 =
𝜋 𝜃 ( 𝒂 𝑡 | 𝒔 𝑡 ) 𝜇 ( 𝒂 𝑡 | 𝒔 𝑡 ) R 𝝓 ( 𝝉 ) (cid:105) .
In practice, we also need to
1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 Samples (Log Scale) 0.
(a) MAML-LVC
SEE ESE
SSE 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 Samples (Log Scale) 0.
(b) MAML-AD
SEE ESE
SSE    5 Step 0.
C o rr (c) LIRPG-LVC 16     5 Step 0.
C o rr (d) LIRPG-AD 16 manually control the level of off-policyness to prevent the variance introduced by the importancesampling process from increasing the estimation variance conversely.
Mutli-step Hessian Estimator.
From the theoretical analysis in Sec.
4.3, we can know that the
Hessian estimation bias can significantly increase the meta-gradient estimation bias in the multi-stepinner-loop setting.
Many low-bias hessian estimator have been proposed in the scope of MAML-RL.
However, the effect of them have never been verified in the general GMRL context.
Here we havea second look at the Low Variance Curvature (LVC) method [28], one can replace the originallog-likelihood with the LVC operator in the policy gradient step.
As such, the Hessian estimator∇
[Equation]
( 𝝓 , 𝜽 ) takes the form: ∇ 𝜃 𝔼 𝝉 ∼ 𝑝 ( 𝝉 ; 𝜽 ) (cid:104)(cid:205) 𝐻 − 1 𝑡 = 0 ∇ 𝜽 𝜋 𝜽 ( 𝒂 𝑡 | 𝒔 𝑡 ) ⊥ 𝜋 𝜽 ( 𝒂 𝑡 | 𝒔 𝑡 ) R 𝝓 ( 𝝉 ) (cid:105) where ⊥ is the stop-gradientoperation which detaches the gradient dependency from the computation graph.
As shown in [28],the LVC operator will ensure an unbiased first-order policy gradient and low-biased low-variancesecond-order policy gradient.
Essentially, this operator only corrects the meta-gradient update andleaves the inner-loop gradient estimation formula untouched.
Experiments
In this section we conduct empirical evaluation of our proposed bias analysis and the proposedmethods to mitigate the biases, and our experiments cover all 3 GMRL fields listed in Table 1.
Inparticular, we conduct a tabular MDP experiment to show the existence of two biases discussed in
Sec.
4 using MAML [10] and LIRPG [41].
In order to show how the proposed methods can mitigatethe compositional bias, we consider a Iterated Prisoner Dilemma (IPD) problem and use LOLA[13] with off-policy corrections in Sec. 4.4; Similarly, we conduct evaluations on Atari games using
MGRL [38] with LVC corrections in Sec.
4.4 to mitigate the Hessian estimation bias.
5.1 Investigate the bias of meta-gradient estimators
Firstly, we conduct experiments to study the bias of meta-gradient estimators using a tabular exampleusing MAML and LIRPG.
To align with existing works in the literature, we adopt the settings ofrandom MDPs in [32] with the focus on meta-gradient estimation.
We refer readers to Appendix J.1.1for more experimental details.
MAML-RL.
Recall that the meta-gradient in MAML is ∇ 𝜽
𝐽 Out ( 𝜽 𝐾 ) .
To control the effect of gradientestimations, we use the three estimators to estimate following three terms: (I) inner-loop policygradient ∇ 𝜽 𝐽 In ( 𝜽 ) ; (II) Jacobian/Hessian ∇ 2 𝜽 𝐽 In ( 𝜽 )
; (III)
outer loop policy gradient ∇ 𝜽 𝐾 𝐽 Out ( 𝜽 𝐾 )
Refer to Appendix J.1.2 for the implementation of decomposing meta-gradient estimation withdifferent estimators.
To show how the estimation is biased, we use a stochastic estimator denoted as S, and exact analyticcalculator denoted as E, for all three derivative terms.
Thus, we can have 7 valid permutations in theexperiment to validate the estimation, where the rest EEE estimator is the exact gradient.
We conduct our ablation studies by comparing the correlation between of meta-gradient with theexact one.
We pick this metric because the bias and variance of meta-gradient are both related withcompositional bias and Hessian bias from Theorem 4.5.
The correlation metric, which is determinedby bias and variance, can show how the final estimation quality is influenced by these two bias terms.
As the quality of stochastic estimators vary from many factors, we conduct this ablation study underextensive combinations of estimation algorithms, learning rates, sample sizes, .etc.
Due to the page [Figure] limit, the results illustrated in Fig.
1(a,b) only include ablation study on sample size and estimationusing exact gradients for (III), the outer-loop policy gradient.
For the rest ablation study and moreevaluation metrics (variance of estimation), refer to Appendix J.1.3 for more details.
We start our evaluation by increasing the sample size.
For simplicity, we only conduct one innerstep here.
In Fig. 1(a), we leverage the LVC [28] estimator.
We can see that a correct inner-looppolicy estimation and/or a Hessian estimation can significantly improve the estimation quality (as
SEE ≈ ESE > SSE in low sample size case).
By increasing the sample size, we can see the gapbetween them is shrinking, which verifies the finding in Lemma 4.4.
In this case the compositionalbias correction shares the same importance with the Hessian bais correction ( SEE ≈ ESE ).
Wealso compare it with the original yet biased gradient estimator of MAML in Fig. 1(b), in which
SEE > ESE in all sample size settings.
In fact, since the gradient estimation is biased, only SEEachieves near 1 .
correlation provided sufficient samples, again confirms the importance of Hessianbias corrections in Sec. 4.2.
LIRPG.
In this setting, we follow the algorithm of intrinsic reward generator presented in [41].
In tabular MDP, we have an additional meta intrinsic reward matrix 𝜙 .
Starting from 𝜽 0 , theinner-loop process takes policy gradient based on the new reward matrix 𝑅 new
= 𝑅 + 𝝓 : 𝜽 𝑖 + 1 =𝜽 𝑖
+ 𝛼 ∇ 𝜽 𝑖 𝐽
In ( 𝜽 𝑖 , 𝜙 ) , 𝑖 ∈ { 0 , 1 ...𝐾 − 1 } .
The meta-gradient estimation of the intrinsic reward matrix∇ 𝝓 𝐽 Out ( 𝜽 𝐾 ) is needed in this case.
Note that in the outer loss we use the original reward matrix𝑅 so the outer loss is 𝐽 Out ( 𝜽 𝐾 ) rather than 𝐽 Out ( 𝝓 , 𝜽 𝐾 ) .
Compared with MAML-RL, the object ofmeta-update (intrinsic matrix) and the object of inner-update (policy parameters) are different, whichhelp us identify the problem mentioned in Sec. 4.2.
In this case, we choose the LVC and AD estimator.
We conduct ablation study on inner-step andsample size shown in Fig.
1(c,d).
With more sample size and less step size, the correlation increasesfor both estimator.
Two important features are: (1) With 1-step inner-loop setting, both estimatorperforms similarly in the correlation.
(2) With multi-step inner-loop setting, LVC based estimator canstill reach relatively high correlation while MAML-biased estimator directly reaches low correlationafter 5-step inner-loop.
The phenomenon shown here corresponds exactly to the Hessian estimationissue we discuss in Sec.
4.2 and the bias issue will be more severe with multi-step inner-loop setting.
5.2 Compositional
bias/off-policy learning in LOLA
In this subsection, we conduct three experiments on Iterated Prisoner Dilemma (IPD) with the LOLAalgorithm to show: (1) The effect brought by different inner/outer estimators.
The effect broughtcompisitional bias (Sec. 4.1) (3) How off-policy correction (Sec. 4.4) can help the LOLA algorithm.
Please refer to Appendix J.2.1, J.2.2 for more experimental setting and results.
Ablation on LOLA-DiCE inner/outer estimation.
We report the result of conducting ablation studyfor different inner/outer-loop estimation of LOLA-DiCE in the Fig. 2(a).
Here the inner-loop estimation refers to ∇ 𝜽 𝐽
In ( 𝝓 , 𝜽 ) while outer-loop estimation refers to ∇ 𝜽 1 𝐽 Out ( 𝝓 , 𝜽 1 ) and ∇ 𝝓 𝐽 Out ( 𝝓 , 𝜽 1 ) .
The return shown in Fig.
2(a) reveals us two findings: 1) The inner-loop gradient estimation plays animportant role for making LOLA work—the default batch size 128 fails while the batch size 1024succeeds.
The outer-loop gradient estimation is also crucial to the performance of LOLA-Exact.
Furthermore, we continue conducting ablation studies on the inner-loop gradient update.
[Figure]
Ablation on compositional bias.
Since the unbiased DiCE estimator is used in LOLA-DiCEalgorithm, there is no Hessian estimation bias in the LOLA algorithm.
Thus, we mainly discuss theproblem of compositional bias brought in Fig. 2(b).
We also apply the implementation in Sec.
5.1 todecompose meta-gradient estimation with different estimators.
Fig. 2(b) show us the ablation studyover compositional bias, which reveals that: compositional bias may decrease the performance andby adding more samples or using analytical solution, the performance can start to improve.
Off-policy DiCE and ablation study.
We use the off-policy learning to conduct inner-loop updateand keep the outer-loop gradient same as before.
By combing DiCE and off-policy learning, wehave off-policy DiCE
𝐽
[Figure]
2 𝑡 ′ )
(cid:19) 𝑅 𝑡 (cid:21) , where 𝜙, 𝜃 refer to thecurrent policy, 𝜇 1 , 𝜇 2 refer the behaviour policy for agent 1 and agent 2, respectively.
H is thetrajectory length and 𝑅 refers to the reward for agent.
Note that the off-policy DiCE here can notonly lower the compositional bias by lowering the first-order policy gradient error, but also helpslower the Hessian variance theoretically.
By the decomposition trick, we conduct experiments bytraversing over all learning settings for (off-off/off-on/on-off/on-on), which are shown at Fig. 2(d).
Comparisons between different settings verify that off-policy DiCE can increase performance byeither lowering the compositional bias, or the hessian variance, or both.
5.3 Multi-step Hessian correction on MGRL
Finally, we conduct experiment over MGRL [38].
See Appendix J.3.1 and J.3.2 for experimentalsettings.
When applying the LVC estimator in MGRL, we get the new inner-loop update equation:𝔼 𝝉 ∼ 𝑝 ( 𝝉 ; 𝜽 ) (cid:104)(cid:205) 𝐻 − 1 𝑡 = 0 ∇ 𝜽 𝜋 𝜽 ( 𝒂 𝑡 | 𝒔 𝑡 ) ⊥ 𝜋 𝜽 ( 𝒂 𝑡 | 𝒔 𝑡 ) (cid:0) 𝑔 𝝓 ( 𝝉 ) − 𝑣 𝜽 ( 𝒔 𝑡 ) (cid:1)(cid:105) , where 𝝓 = ( 𝛾, 𝜆 ) refer to meta-paramters and 𝜃refers to the RL policy, 𝑔 𝝓 ( 𝜏 ) denotes 𝜆 -return, 𝑣 𝜽 ( 𝑠 𝑡 )
denotes value prediction.
We conductexperiment on eight environments of Atari games.
We follow previous work [4] to use the "discard"strategy in which we conduct multiple virtual inner-loop updates for meta gradient estimation.
Thisstrategy is designed to keep the inner learning update unchanged.
In Fig.
(3), we show five environments comparing three variants of algorithm: 1) Baseline Advantage
Actor-critic(A2C) algorithm [25]; 2) 3-step MGRL + A2C; 4)
3-step MGRL + A2C + LVC correction.
The "3-step" means we take 3 inner-loop RL virtual updates for calculating meta-gradient.
Refer to
Appendix J.3.3 for experimental results on all eight environments.
Compared with 3-step MGRL,the MGRL with LVC correction can substantially improves the performance, which validates theeffectiveness of the multi-step Hessian correction in Sec.
4.4 for handling meta-gradient estimationbias and bring in better hyperparameter-tuning in RL.
Note that the fact that A2C algorithms canachieve better results compared with 3-step MGRL is consistent with the results in [38].
Conclusion
In this paper, we introduce a unified framework for studying generic meta-gradient estimations ingradient-based Meta-RL.
Based on this framework, we offer two theoretical insights that 1) thecompositional bias has an upper bound of O (cid:0)
𝐾𝛼 𝐾 ˆ 𝜎
In | 𝜏 | − 0 .
5 (cid:1) with respect to the inner-loop updatestep 𝐾 , the learning rate 𝛼 , the estimate variance ˆ 𝜎
In and the sample size | 𝜏 | , and 2) the multi-step
Hessian bias ˆ Δ 𝐻 has a polynomial impact of O (cid:0) ( 𝐾 − 1 )( ˆ Δ 𝐻 ) 𝐾 − 1 (cid:1) .
To validate our theoreticaldiscoveries, we conduct a comprehensive list of ablation studies.
Empirical results over tabular MDP,
LOLA-DiCE and MGRL validate our theories and the effectiveness of correction methods.
Webelieve our work can inspire more future work on unbiased meta-gradient estimations in GMRL.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Appendix B.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Sec. (b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix G and Appendix H
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e g., data splits, hyperparameters
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] , See Fig.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix J4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]