A Unifying Framework of Off-Policy General Value Function Evaluation
Anonymous Author(s)
AfﬁliationAddress email
Abstract
General Value Function (GVF) is a powerful tool to represent both the predictiveand retrospective knowledge in reinforcement learning (RL).
In practice, oftenmultiple interrelated GVFs need to be evaluated jointly with pre-collected off-policysamples.
In the literature, the gradient temporal difference (GTD) learning methodhas been adopted to evaluate GVFs in the off-policy setting, but such an approachmay suffer from a large estimation error even if the function approximation classis sufﬁciently expressive.
Moreover, none of the previous work have formallyestablished the convergence guarantee to the ground truth GVFs under the functionapproximation settings.
In this paper, we address both issues through the lens of aclass of GVFs with causal ﬁltering, which cover a wide range of RL applicationssuch as reward variance, value gradient, cost in anomaly detection, stationarydistribution gradient, etc.
We propose a new algorithm called GenTD for offpolicy GVFs evaluation and show that GenTD learns multiple interrelated multidimensional GVFs as efﬁciently as a single canonical scalar value function.
Wefurther show that unlike GTD, the learned GVFs by GenTD are guaranteed toconverge to the ground truth GVFs as long as the function approximation poweris sufﬁciently large.
To our best knowledge, GenTD is the ﬁrst off-policy GVFevaluation algorithm that has global optimality guarantee.
Introduction
The value function, which represents the expected accumulation of reward [44], serves as a reliableperformance metric of policy in the reinforcement learning (RL) tasks [43, 23].
In many RLapplications, however, looking at only the value function is not enough.
For example, in the risksensitive domains such as health care and ﬁnancial assets, the variance of "reward-to-go" ratherthan the value function, i e., the mean of "reward-to-go", is a more suitable performance metric.
Asanother example, to obtain a variance-reduced or bias-reduced policy gradient estimator [14, 62, 18],in addition to the value function, the information of "gradient of value function" is also required.
Moreover, in continuous control domain with differentiable and deterministic policy, the computationof policy gradient is only possible through "action/state-value gradient" [39, 8, 12], etc.
All theaforementioned metrics can be viewed as predicative knowledge of certain multiple intercorrelatedcumulative "signals" (possibly high-dimensional, e.g., the gradient of value function), and thusnaturally fall into the framework of forward GVFs (refers to forward general value functions)[49, 58, 33, 34] (see Section 2.1 for the formal deﬁnition).
One typical approach to evaluate GVFs,is to learn from samples that pre-collected from one or more behavior policies, which yields anoff-policy method.
In practice, multiple forward GVFs are usually evaluated jointly at the same timedue to their interrelationships [38, 34].
In contrast to forward GVFs deﬁned based on predictive knowledge, the backward GVFs representsretrospective knowledge, which captures the accumulation of signals from the past to the present time [71] (see Section 2.2 for the formal deﬁnition).
Although the concept of the backward GVFshas not been formally proposed until very recently [71], it is rooted in a number of important RLapplications such as anomaly detection [71], emphatic weight learning [47, 71] and evaluation ofgradient of logarithmic stationary distribution [26, 62, 18].
Different from the forward GVFs, forwhich the Bellman operator can be deﬁned independently from the sampling distribution [38, 43, 45],the Bellman operator of the backward GVFs is only valid if the sampling exactly follows the on-policystationary distribution [71].
Due to such a reason, off-policy evaluation of the backward GVFs ismuch more challenging than that of the forward GVFs.
In general, due to the high dimensionality and intercorrelation, it is very challenging to evaluatemultiple GVFs simultaneously with standard policy evaluation approaches [56, 46, 23].
In previousstudies, the gradient temporal difference (GTD) learning [46, 23], one of the most popular off-policymethods in value function evaluation, has been adopted to solve both the forward and backward
GVFs evaluation problems [49, 38, 70].
GTD adopts the mean squared projected Bellman error(MSPBE) as its optimization objective and takes the expectation over the behavior policy, which doesnot exactly reﬂect the desirable evaluation under the target policy.
As a result, GTD can encounterserious issues in GVFs evaluation problems.
First , the optimal point to which GTD converges can befar away from the ground truth value of GVFs.
It becomes worse when multiple GVFs are evaluatedsimultaneously, because the error of one GVFs evaluation can be further ampliﬁed across other
GVFs’ evaluation due to their inherent correlations.
In the literature, no provable bound has beenestablished on such an error, which can, in fact, be unbounded for some cases (see Example 1 in[19]).
Second , for high-dimensional GVFs evaluations, the landscape geometry of the GTD objectivefunction can be ill-conditioned [23], which could slow down the convergence of GTD signiﬁcantly.
As demonstrated by our empirical results in Section 5, GTD can suffer from both the large estimationerror and the slow convergence rate, which further suggests that GTD may not be a good choice for
GVFs evaluation tasks.
This motivates our paper to address the following question:
• Can we design a new off-policy approach for multiple interrelated and high-dimensional GVFsevaluation problems, which is guaranteed to converge fast and converge to the ground truth GVFs?
Our Contributions.
In this paper, we investigate the problem of evaluating multiple interrelated
GVFs jointly.
Rather than studying different GVFs on a case-by-case basis, we explore the classof "GVFs with causal ﬁltering", which captures a common structural feature shared by GVFs ina wide range of RL applications (see Appendix C).
(a) We prove that both forward and backward
GVFs with causal ﬁltering are the unique ﬁxed point of their corresponding general Bellman operator(GBO) (deﬁned for multiple high-dimensional GVFs), which is shown to have a contraction propertywith respect to a properly constructed norm metric.
(b) Based on such a property of GVFs, wepropose a new algorithm GenTD to solve off-policy GVFs evaluation problem.
GenTD introduces adensity ratio to adjust the behavior distribution and further incorporates a policy-agnostic approach
GenDICE/GradientDICE [66, 71] for estimating the density ratio jointly with GVF evaluation.
(c) Inthe linear function approximation setting, we show that GenTD converges to the globally optimalpoint at the rate of O (1 /T ) , with conditional number independent from the dimension of GVFs.
Such a result implies that GenTD learns multiple interrelated possibly high-dimensional GVFs asefﬁciently as TD learning for a single canonical scalar value function.
(d) We further show thatunlike GTD, GenTD is guaranteed to approximate the ground truth GVFs well as long as the functionexpressive power is sufﬁciently large.
To our best knowledge, GenTD is
Related Work.
The forward GVF was ﬁrst introduced in [49] to represent a set of accumulationof general signals with possibly time-varying discount factors.
The forward GVF was later usedto represent a set of interrelated predictions [38, 9, 42, 24, 34].
It has been observed that some RLmetrics such as variance, gradient of value function, state/action value gradient can also be viewed asforward GVFs [52, 14, 62, 18, 39, 49, 58, 33, 8, 5].
In previous works, both TD learning and GTDhave been used to evaluate forward GVFs in the on- and off-policy settings [49, 38, 34], respectively.
A more comprehensive review of studies of forward GVFs has been provided in [36].
The backward
GVF was formally deﬁned in [71].
Some previous works have also considered metrics that can berepresented as accumulations of signals in the reverse time direction, such as emphatic weighting,page ranking cost, and derivative of logarithmic stationary distribution [67, 69, 26, 64, 10].
Anothertrack of research has focused on evaluation of a general scalar function in the off-policy setting, a.k.a off-policy evaluation (OPE)
However, since the focus of this paper is on the evaluation ofmultiple high-dimensional GVFs, results in OPE are not directly comparable with ours.
The theoretical studies of off-policy GVFs evaluation algorithms are rather limited.
So far, only theasymptotic convergence guarantee (without the convergence rate characterization) of GTD has beenestablished in both the forward and backward GVFs evaluation settings [38, 71].
The convergencerate of GTD has only been established in [63, 7, 17, 6, 59, 21] for the simple canonical value functionevaluation setting, which is a special case of forward GVFs.
However, as pointed out in [19, 10, 27],the optimal point of GTD may suffer from possibly unbounded approximation error, which is notdesirable in practice.
In contrast, we propose a new off-policy GVFs evaluation algorithm, whichcan solve a wide range of forward and backward GVFs evaluation problems, with convergence ratecharacterization and guaranteed optimality with respect to the ground truth GVFs value.
We note that the contraction property of GBO for forward GVFs has also been investigated in [34]with a structure called "acyclic graph", which is similar to "causal ﬁltering" in our paper (see thefootnote comment for Proposition 1).
However, the results of backward GVFs are established inour work for the ﬁrst time.
The focus of this paper is on the ﬁnite time performance and optimalityguarantee for a new off-policy GVFs evaluation algorithm GenTD, which was not studied in [34].
Markov Decision Process and General Value Function
We consider an inﬁnite-horizon Markov Decision Process (MDP) with a state space S , an actionspace A , a reward function r : S ×
A → R , a transition kernel P : S × S ×
A → [0 , 1] , a discountedfactor γ ∈ (0 , 1) , and an initial distribution µ 0 : S → [0 , 1] .
An policy π ( a | s ) is the probabilityof taking action a at state s .
At time step t , an agent at a state s t selects an action a t according toπ ( ·| s t ) , receives a reward r ( s t , a t ) , and transits to state s t +1 according to P ( ·| s t , a t ) .
The state-actiontransition kernel is deﬁned as P π ∈ R |S||A|×|S||A| , in which P π (( s, a ) , ( s (cid:48) , a (cid:48) ))
=
P ( s (cid:48) | s, a ) π ( a (cid:48) | s (cid:48) ) .
When the MDP is ergodic, we deﬁne µ π as the state-action stationary distribution which satisﬁes:µ (cid:62)
π P π = µ (cid:62) π .
For such an MDP, we deﬁne the discounted accumulation of reward as the "rewardto-go": J π = (cid:80) ∞ t =
γ t r ( s t , a t ) .
The state-action value function (i.e., Q-function) is deﬁned as
Q π ( s, a ) =
E [ J π | ( s 0 , a 0 ) =
( s, a )] , and the state value function (i.e., V-function) is deﬁned as
V π ( s ) =
E [ Q π ( s, a ) | s ] .
Note that Q π ( s, a ) satisﬁes the following Bellman equation
[Equation]
where T π is the Bellman operator, and Q π and R ∈ R |S||A| are vectors obtained via stacking
Q π ( s, a ) and r ( s, a ) over state-action space S × A .
We introduce a function of ( s, a ) (possibly inthe vector form) as v ( s, a ) ∈ R d ( d ≥ 1 ).
Consider a distribution ξ ( · ) over S × A .
We deﬁne theξ –norm of v ∈ R d |S||A| as (cid:107) v (cid:107) ξ = (cid:113)(cid:80) ( s,a ) ξ ( s, a ) (cid:107) v ( s, a ) (cid:107) 22 , where v is obtained by stackingthe function v ( s, a ) over S × A .
It has been proved that T π is γ –contraction in µ π –norm, i.e.,(cid:107)T π v − T π v (cid:48)
(cid:107) µ π ≤ γ (cid:107) v − v (cid:48)
(cid:107) µ π and Q π is the unique ﬁxed point of T π [45, 43, 56].
In the sequel,we denote I d as the identity matrix with the dimension d and ⊗ as the Kronecker product.
Wefurther deﬁne U π =
diag
( U π, 1 , · · · , U π,k ) , in which U π,i = diag ( µ π ) ⊗ I
d i for i = { 1 , · · · , k } , and
P π = diag ( P π, 1 , · · · , P π,k ) , in which P π,i =
P π ⊗ I d i .
2.1 Forward General Value Function
Consider a set of the state-action general value functions (GVFs) G π =
[ G (cid:62) π, 1 , · · · , G (cid:62) π,k ] (cid:62) , whereeach GVF G π,i is deﬁned as the accumulation of a corresponding signal C i ( s, a
) ∈ R d i givenby G π,i ( s, a )
=
E (cid:2)(cid:80) ∞ t =
γ ti C i ( s t , a t ) (cid:12)(cid:12) ( s 0 , a 0 ) =
( s, a ) , π (cid:3) , where γ i ∈ (0 , 1) is a discount factorassociated with C i .
Since C i ( s, a ) ∈ R
d i can be high-dimensional, G π,i ( s, a ) can also be highdimensional for each ( s, a ) .
Clearly, the Q-function is a special GVF associated with a scalar signal.
Since G π is deﬁned as the accumulation of the signal C i in a forward direction from the current timestep t to the future ∞ , we call G π as "forward GVF".
In many RL applications, we are often interested in the case that G π of GVFs have progressivedependence [49], i.e., each C i ( s, a ) (associated with G π,i ) depends on the lower-indexed valuefunctions G π, 1 , · · · , G π,i − 1 in the set.
As a concrete example, suppose the policy is parametrizedby a smooth function π w , where the parameter w ∈ R d w .
In addition to the Q-function Q π , the gradient ∇ w Q π ( s, a ) of the Q-function w.r.t w arises as a GVF of interest in several applications.
In such a case, G π, 1 = Q π and G π, 2 = ∇ w Q π .
It has been shown in [5] that the reward C 2 ( s, a )associated with ∇ w Q π is given by C 2 ( s, a )
= γ E
[ Q π ( s (cid:48) , a (cid:48) ) ∇ w log( π w ( s (cid:48) , a (cid:48) ))
| s, a ] , which dependson the lower-indexed G π, 1 = Q π .
Hence, such deﬁned GVFs vector has progressive dependence.
Appendix C provides further details and more examples in RL.
More formally, we refer to thisstructure of forward GVF with progressive dependence as casual ﬁltering as deﬁned below.
Note thata similar structure was called acyclic graph in [34].
Deﬁnition 1 (Forward GVF with causal ﬁltering) .
For a given policy π , a forward GVF G π =[ G (cid:62) π, 1 , · · · , G (cid:62) π,k ] (cid:62) with causal ﬁltering are associated with signals satisfying
[Equation]
where C i and G π,j are obtained by respectively stacking C i ( s, a
) ∈ R d i and G π,j ( s, a ) ∈ R d i over
S × A , B i ∈ R
d i |S||A| is an observable signal, and the coefﬁcient matrix A i,j ∈ R d i |S||A|× d j |S||A|captures how the j -th GVF G π,j affects the i -th accumulation signal C i .
Further, B i and A i,j arebounded to ensure G π,i to be well deﬁned.
Deﬁnition 1 indicates that all GVFs are interrelated with a causal ﬁltering structure, i.e., each signal
C i is a linear function of all lower-indexed G π,l for 1 ≤ l < i .
Deﬁnition 1 also implies that theforward GVF
G π =
[ G (cid:62) π, 1 , · · · , G (cid:62) π,k ] (cid:62) with causal ﬁltering satisﬁes the following lower-triangular
Bellman equation given by
[Equation]
where T G,π denotes the forward general Bellman operator (GBO), B = (cid:2) B (cid:62) 1 , · · · , B (cid:62) k (cid:3) (cid:62) and
[Equation]
where ¯ P π,i =
[ P π ⊗ I d i ] .
Clearly, the canonical value function Q π and Bellman operator T π deﬁnedin eq.
(1) is a special case of G π and T G,π deﬁned in eq.
2.2 Backward General Value Function
In contrast to the forward GVF deﬁned in the last section, which represents the predictive knowledge ,in some RL scenarios, we also want to capture the retrospective knowledge , which represents theaccumulation of signals that have been collected from the past.
Consider a set of GVFs ˆ G π =[ ˆ G (cid:62) π, 1 , · · · , ˆ G (cid:62) π,k ] (cid:62) , where each GVF ˆ G π,i is deﬁned as the backward accumulation of a vectorsignal
ˆ C i ( s, a ) ∈ R d i given by ˆ G π,i ( s, a ) = E (cid:2) (cid:80) 0 t =
−∞ γ − t
i ˆ C i ( s t , a t ) (cid:12)(cid:12) ( s 0 , a 0 )
=
( s, a ) , π (cid:3) .
Todistinguish from the forward GVF G π,i deﬁned in Section 2.1, we denote ˆ G π,i as the backward GVF.
For general purpose, we also consider the causal ﬁltering setting for ˆ G π , in which each ˆ C i ( s, a )depends on the lower-indexed value functions ˆ G π, 1 , · · · , ˆ G π,i − 1 in the set.
We deﬁne the backward
GVF with causal ﬁltering as follows.
Deﬁnition 2 (Backward GVF with causal ﬁltering) .
For a given policy π , a backward GVF ˆ G π =[
ˆ G π, 1 , · · · , ˆ G π,k ] with causal ﬁltering are associated with signals satisfying
[Equation]
≤
[Equation] ≤ k [Equation] where ˆ C i and ˆ G π,j are obtained by respectively stacking ˆ C i ( s, a ) ∈
R d i and ˆ G π,j ( s, a ) ∈ R d i over
S × A , B i ∈ R
d i |S||A| is an observable signal, and the coefﬁcient matrix A i,j ∈ R d i |S||A|× d j |S||A|captures how the j -th GVF ˆ G π,j affects the i -th accumulation signal ˆ C i .
Further, B i and A i,j arebounded to ensure ˆ G π,i to be well deﬁned.
For an ergodic MDP that starts from −∞ , we have ( s t − 1 , a t − 1 ) ∼ µ π ( · ) , ( s t , a t ) ∼ P π ( ·| s t − 1 , a t − 1 ) ,and ( s t , a t ) ∼ µ π ( · ) for all −∞ < t < ∞ .
The Bayes’ theorem implies that
[Equation]
The reverse conditional probability in eq.
(3) together with the deﬁnition of backward GVF in
Deﬁnition 2 implies that the backward GVFs ˆ G π =
[ ˆ G (cid:62) π, 1 , · · · , ˆ G
[Equation]
k ] [Equation] with causal ﬁltering satisﬁes
[Equation]
(4) where ˆ T G,π denotes the backward GBO, B = (cid:2) B (cid:62) 1 , · · · , B (cid:62) k (cid:3) (cid:62) , andˆ M π =   γ
1 ˆ P π, · · ·
A 2 ,γ 2 ˆ P π,· · · ...
A k,
A k,· · · γ k ˆ P π,k   , where ˆ P π,i =
U − 1 π,i
[ P π ⊗ I
d i ] U π,i .
Off-Policy Evaluation of GVFs3.1 Problem Formulation
In this paper, we study the GVFs evaluation problem for a target policy π .
We focus on the behavioragnostic off-policy setting, in which we have access only to samples generated from an off-policy(i.e., a behavior policy) with the distribution D , i.e., ( s j , a j , B j , s (cid:48)
j ) ∼ D ( j > 0) .
Speciﬁcally, thestate-action pair ( s j , a j ) is sampled from a possibly unknown distribution D ( · ) :
S ×
A → [0 , 1] ,
B j =
[ B 1 ( s j , a j ) , · · · , B k ( s j , a j )] is an observable signal vector, and the successor state s (cid:48)
i issampled from P ( ·| s i , a i ) .
Our goal is to design an efﬁcient algorithm to estimate G π (or ˆ G π ) giventhe sample set { ( s j , a j , B j , s (cid:48) j ) } j> 0 .
We make the following dataset coverage assumption.
Assumption 1.
We assume that D ( s, a ) > 0 for all ( s, a ) ∈ S × A .
3.2 Linear Function Approximation
When |S| is large, a linear function can be used to approximate the GVF: G π,i ( s, a ) ≈
G π,i ( θ i ; s, a ) =
θ (cid:62) i φ i ( s, a ) =
[ φ i ( s, a ) (cid:62) ⊗ I
d i ] vec ( θ (cid:62) i ) , where φ i ( s, a ) ∈ R K
i is the feature vector,and θ
i ∈
R K i ×
d i is a learnable weight matrix.
In the sequel, we omit π in G π,i and use the notation
G i .
We make the following assumption for linear feature φ , which is standard in linear functionapproximation setting [46, 23, 56].
Assumption 2.
We assume that (cid:107) φ i ( s, a ) (cid:107) 2 ≤ 1 for all i = 1 , · · · , k and ( s, a ) ∈ S × A .
The linear approximation can then be written as G i ( θ i ) =
[Φ i ⊗ I
d i ] vec ( θ (cid:62) i ) , where Φ i isthe base matrix obtained by stacking φ i ( s, a ) (cid:62) over S × A .
To ensure the uniqueness ofthe solution θ i , we assume that Φ i has linearly independent columns.
The joint vector of
GVFs can be denoted as [ G (cid:62) 1 ( θ 1 ) , · · · , G (cid:62)
k
( θ k )] (cid:62) , which is captured by the joint parametersθ
[Equation] [Equation]
k i =
1 K i d i .
Then the function approximation of GVFs can bewritten more compactly as G ( θ ) =
Φ θ , where Φ = diag([Φ 1 ⊗ I d 1 ] , · · · , [Φ k ⊗ I d k ]) .
For each ( s, a ) ,the linear function approximation associated with each ( s, a ) can be written as G ( θ ; s, a ) =
φ ( s, a ) θ ,where φ ( s, a ) =
diag([ φ 1 ( s, a ) (cid:62) ⊗ I d 1 ] , · · · , [ φ k ( s, a ) (cid:62) ⊗ I d k ]) .
We deﬁne the linear function spacespanned by the columns of the feature matrix Φ as F Φ = { Φ θ | θ ∈ R θ } , in which R θ is a convex set.
Given the function class F Φ , the evaluation problem of GVFs amounts to searching for a parameterθ ∗ ∈ R θ such that G ( θ ∗ ) approximates G π (or ˆ G π ) well.
In the sequel, we use ¯ T G,π to represent
T G,π or ˆ T G,π , interchangeably, based on the context.
3.3 A New Off-policy GVF Evaluation Approach
Drawbacks of GTD.
In previous works, the gradient TD (GTD) method [46, 23] has been usedfor policy evaluation (including GVF evaluation) in the off-policy setting [38, 70, 71, 62].
GTDadopts the Mean Squared Projected Bellman Error (MSPBE) for GVF evaluation with linear functionapproximation, which is given by
[Equation]
where Γ F Φ ,D denotes the projection operator onto the space F Φ w.r.t.
the (cid:107)·(cid:107) D –norm, i.e., for any vector function f ( s, a ) of ( s, a )
, we have Γ F Φ ,d f = G ( θ f ) , in which θ f = argmin θ ∈ R θ (cid:107)
f − G ( θ ) (cid:107) D .
One drawback of GTD is that the expectation in the objective function is taken over the off-policysampling distribution D ( · ) , which does not exactly reﬂect the desirable evaluation under the targetpolicy.
As the result, the optimal point of GTD ( ˆ θ ∗ ) can still have a large approximation error withrespect to the ground truth value of GVF, even if the approximation function class is arbitrarilyexpressive.
More detailed discussion about GTD is provided in Appendix D.
Generalized Temporal Difference (GenTD) Learning.
In this work, we propose a novel uniﬁedapproach to evaluate both the forward and backward GVFs in the off-policy setting, which we referto as generalized temporal difference (GenTD) learning.
Speciﬁcally, we aim to learn θ ∗ for GVFevaluation by minimizing the mean-squared projected general Bellman error (MSPGBE) deﬁned as
[Equation]
(6) where recall that ¯ T G,π represents the GBO of either forward or backward GVFs.
In contrast to GTD,the objective function in eq.
(6) takes the expectation over the stationary distribution µ π of the targetdistribution, which precisely captures the desired goal of GVF evaluation under the target policy.
Onthe other hand, such an objective does cause implementation challenge, because the data samples aregenerated by the behavior policy, so that estimators based on such data directly can incur a large biaserror.
To solve such an issue, we will apply the density ratio ρ ( s, a ) = µ π ( s, a ) /D
( s, a ) to adjust thedistribution and further adopt the GenDICE/GradientDICE method proposed in [66, 68] to estimateρ ( s, a ) during the execution of the algorithm.
To describe our algorithm GenTD (see Algorithm 1), we ﬁrst note that eq.
(6) implies the followingoptimality condition for θ ∗ and all f ∈ F Φ ,
[Equation]
or equivalently
[Equation]
where g ( θ ) =
Φ (cid:62) U π ( G ( θ ) − ¯ T G,π G ( θ )) .
The variational inequality theory in (Chapter 3 [20])suggests that under an appropriately chosen stepsize α t , the update θ t +1 =
Γ R θ ( θ t − α t g ( θ t ))converges to the optimal point θ ∗ , where Γ R θ denotes the projection operator onto the set R θin terms of the Euclidean norm.
However, since it is intractable to explicitly compute g ( θ ) inpractice, we usually estimate g ( θ ) using random samples.
In the off-policy setting, consider a samplex =
( s, a, s (cid:48) , a (cid:48) ) , in which ( s, a ) ∼ D ( · ) , s (cid:48) ∼ P ( ·| s, a ) , and a (cid:48) ∼ π ( ·| s (cid:48) ) , we can formulate thefollowing update rule:
[Equation]
where ˆ ρ ( s, a ) is an approximation of the density ratio ρ ( s, a ) =
µ π ( s, a ) /D
( s, a ) , g ( x, θ ) =
− φ ( s, a ) (cid:62) δ ( x, θ ) for forward GVFs and g ( x, θ ) = − φ ( s (cid:48) , a (cid:48) )
(cid:62) δ ( x, θ ) for backward GVFs, whereδ ( x, θ ) is the temporal difference error deﬁned as δ ( x, θ )
=
B ( s, a ) + m ( x ) φ ( s (cid:48) , a (cid:48) )
θ − φ ( s, a ) θ forforward GVFs, and δ ( x, θ ) =
B ( s (cid:48) , a (cid:48) )
+ ˆ m ( x ) φ ( s, a ) θ − φ ( s (cid:48) , a (cid:48) )
θ for backward GVFs.
Here m andˆ m are matrices that capture the correlations between difference estimations in forward and backward
GVFs evaluation settings, respectively.
Here we adopt the GenDICE/GradientDICE method thatproposed in [66, 68] to learn ρ ( s, a ) .
In previous works, GenDIC/GradientDICE has only been usedfor estimating the scalar value J π = E µ π
[ r ( s, a )] in the off-policy setting [66, 71, 54].
Our work isthe ﬁrst to adapt this method to solve the more challenging off-policy GVFs evaluation problem.
Learning Density Ratio.
GenDICE/GradientDICE estimates the density ratio ρ ( s, a ) via solvingthe following min-max problem [66, 68]:
[Equation]
We parameterize both ρ and f by linear function with linearly independent features ψ ∈ R d ρ , i.e.,ˆ ρ ( s, a ; w ρ ) =
ψ ( s, a ) (cid:62) w ρ
and ˆ f ( s, a ; w f )
=
ψ ( s, a ) (cid:62) w f for all ( s, a ) ∈ S × A .
To guaranteesthe stability of the density ratio learning, we assume that the matrix A
= E D· π [ ψ ( ψ − ψ (cid:48) )
(cid:62) ] is
Initialize: Approximator parameters w f, 0 , w ρ, 0 and θ 0 for t = 0 , · · · , T − 1 do
Obtain sample ( s t , a t , C t , s (cid:48) t ) ∼ D d and a (cid:48)
t ∼ π ( ·| s (cid:48) t ) ¯ δ t = ψ (cid:62) t θ ρ,t ( ψ (cid:48) t − ψ t ) η t
[Equation]
1 = w ρ,t
[Equation]
t ( ψ (cid:62)
[Equation] [Equation]
η t )
w
[Equation] [Equation]
=
[Equation] [Equation] [Equation]
(cid:62)
[Equation] [Equation]
t )
w
[Equation]
(cid:0)
[Equation]
[Equation]
[Equation]
(cid:48)(cid:62)
[Equation] [Equation]
(cid:62) t w f,
[Equation]
t
[Equation]
(cid:1)
θ t +
=
Γ R θ (cid:0) θ t − α t
[ w (cid:62) ρ,t ψ ( s t , a t )]
g ( x t , θ t ) (cid:1)
Forward GVF: g ( x, θ )
= − φ ( s, a ) (cid:62) ( B ( s, a ) + m ( x ) φ ( s (cid:48) , a (cid:48) )
θ − φ ( s, a ) θ )
Backward GVF: g ( x, θ )
= − φ ( s (cid:48) , a (cid:48) )
(cid:62) ( B ( s (cid:48) , a (cid:48) )
+ ˆ m ( x ) φ ( s, a ) θ − φ ( s (cid:48) , a (cid:48) ) θ ) end for non-singular.
Note that this assumption can be removed by adding an l 2 –regularizer in eq.
In
GenTD (see Algorithm 1), we estimate the density ratio via updating the parameter w ρ,t iteratively.
The density estimator ˆ ρ ( s t , a t ; w ρ t ) =
ψ ( s t , a t ) (cid:62) w ρ t is then used to reweight the update g ( x t , θ t ) .
Comparison between GenTD and GTD.
Compared with GTD, our GenTD has the following twoadvantages.
First, since GTD does not adjust the distribution mismatch of sampling, the optimal pointof GTD can suffer from large approximation error with respect to the ground truth GVFs even withhighly expressive function classes.
In contrast, the optimum of GenTD is guaranteed to approximatethe ground truth GVFs well with sufﬁciently expressive function classes.
Second, GTD needs toupdate a high-dimensional auxiliary parameter w simultaneously with θ to stabilize the convergence,where w ∈ R (cid:80) ki
=1 K i d
i has the same dimension as θ ∈ R (cid:80) ki
=1 K i d i (note that (cid:80)
ki =1 K i d i can be verylarge in the high dimensional regime or when the number of GVFs k is very large).
Such an updateof w can be very costly.
In contrast, GenTD introduces only low-dimensional auxiliary parameters[ w ρ , w f , η ] ∈ R 2 d
ρ +1 for density ratio estimation, which is more efﬁcient than GTD since d ρ couldbe much smaller than (cid:80) ki
=1 K i d i .
Main Theorems
In this section, we develop the ﬁnite-time convergence rate for our Off-GenTD algorithm.
To thisend, we ﬁrst want to establish a certain contraction property for the general Bellman operator ofinterest here.
Although the contraction property has been proven in the canonical value functionsettings [56, 71], it is unclear whether such a property still holds for multiple multi-dimensionaland interrelated GVFs.
We will next establish that such a property still holds for both forward andbackward GVFs with causal ﬁltering, but needs to be under a properly chosen norm.
Consider the GVFs vector
[Equation]
, G (cid:62) π,k ] (cid:62) .
We deﬁne a norm (cid:107)·(cid:107) µ π ,α associated with aweighting vector
α =
[ α 1 , · · · , α k ] ∈ ∆ k , where ∆ k denotes the simplex in k -dimensional space,as (cid:107) G π (cid:107) µ π ,α = (cid:80) k
i =
α i (cid:107) G π,i (cid:107) µ π , where 0 < α i ≤ 1 for all i and (cid:80) k i =1 α i = 1 .
We also deﬁneγ max := max i =1 ··· ,k γ i , which is strictly less than 1 .
Proposition 1 (Contraction of Forward/Backward GBO) .
For any G π , G (cid:48) π ∈ R |S||A| (cid:80) ki =1 K i d i ,there exists a weighting vector
α such that
[Equation]
where T G,π can be either T G,π (forward GBO, eq.
(2) ) or ˆ T G,π (backward GBO, eq.
Despite the correlations between GVFs, Proposition 1 shows that the contraction property is stillpreserved under a properly chosen norm for T G,π and ˆ T G,π in forward and backward GVF settings,respectively.
The norm can vary for different GVFs.
Proposition 1 also implies that both forward andbackward GVFs ( G π and ˆ G π ) can be identiﬁed as unique ﬁxed point of their corresponding GBOs.
Based on Proposition 1, we next establish the monotonicity property for our GenTD algorithm, if ittakes the population update g ( θ ) =
Φ (cid:62) U π ( G (
[Equation]
Proposition 2 (Monotonicity) .
Suppose Assumption 1 &2 hold.
Consider the globally optimal pointθ ∗ deﬁned in eq.
There exists a constant λ G such that for all θ ∈ R
θ , we have
[Equation]
) where λ G := (1 − γ max ) min 1 ≤ i ≤ k ζ i and ζ i :=
λ min (Φ (cid:62) i
U π Φ i ) .
Proposition 2 implies the contraction property of g ( θ ) .
It guarantees that θ moves towards a globallyoptimal point θ ∗ if it is updated along the direction − g ( θ ) .
Proposition 2 generalizes the monotonicityproperty to a much broader class of interrelated and multi-dimensional GVF evaluation, which isfar more beyond TD learning for the value function evaluation studied in [56, 71].
The followingtheorem characterizes the convergence rate of GenTD.
Theorem 1.
Suppose Assumption 1 &2 hold.
Consider the GenTD update in Algorithm 1.
Let thestepsize α t =
Θ( t − 1 ) and β t =
Θ( t − 1 ) .
We have
[Equation]
(12) where ε ρ = (cid:113) E D· π [ˆ ρ ( s, a ; w ∗ ρ ) − ρ ( s, a )] 2 is the approximation error introduced by the densityratio learning, with w ∗ ρ deﬁned in eq.
Theorem 1 shows that GenTD converges to the globally optimal point θ ∗ at a rate O
(1 /T ) .
Theconvergence speed of θ also depends on the conditional number λ G , where the converge becomesfaster as λ G increases.
Speciﬁcally, the R.H.S. of eq.
(12) consists of three terms.
The ﬁrst termcorresponds to the initialization error, which delays as fast as O (1 /T 2 ) .
The second term correspondsto the variance error, which dominates the convergence rate of GenTD to be O (1 /T ) .
The last termcorresponds to a non-vanishing optimality gap, which is introduced by the function approximationerror in the density ratio estimation, and decreases as the expressive power of the approximationfunction class { ˆ ρ ( w ρ ) : w ρ ∈ R ρ } increases.
For more discussion about this approximation error,please refer to [66, 68].
The convergence analysis of GenTD is more challenging than that of
TD learning [2, 7, 41] and GTD [63, 17], as we need to handle an additional approximation errorintroduced by the dynamically changing density ratio estimator ˆ ρ ( w ρ t ) .
Theorem 1 establishes the convergence of GenTD to the globally optimal point θ ∗ of the objectivefunction in eq.
(6), which provides the value estimation G ( θ ∗ ) for the GVFs.
We are then interestedin characterizing how close such an estimation is to the ground truth GVF G π , which is our ultimategoal of evaluation.
We characterize this in the following theorem.
Theorem 2 (Convergence of GenTD to Ground Truth) .
Consider θ ∗ deﬁned in eq.
Suppose thesame conditions in Proposition 1 & 2 hold.
We have
[Equation]
Theorem 2 indicates that the distance between the optimal estimation G ( θ ∗ ) and the true GVF G πis upper bounded by the approximation error of the function class F Φ for the ground truth GVF
G π (note that Γ F Φ ,µ π G π denotes the projection of G π to the function approximation class F Φ ).
Hence, Theorem 2 guarantees that G ( θ ∗ ) can be as close as possible to the true GVF G π , as longas the function class F Φ is sufﬁciently expressive.
In particular, if F Φ is complete, i.e., there exists
G θ ∈ F Φ such that G θ = G π , then GenTD is guaranteed to converge exactly to the ground truth G π .
Note that Theorem 2 is the ﬁrst result of such a type developed for both forward and backward GVFs.
Comparison between GenTD and GTD.
If F Φ is complete, GTD performs similarly to GenTDand is guaranteed to converge to the ground truth G π (see Appendix D.2 for the proof).
The majordifference between GenTD and GTD occurs when F Φ is not complete.
In such a case, our GenTDstill maintains the desirable performance as guaranteed by Theorem 2, but the optimal point of
GTD (i.e., ˆ θ ∗ in eq.
(5)) does not have guaranteed convergence to the ground truth.
As shown in[19, 10, 27], even in the value function evaluation setting (a special case of forward GVF evaluation)the approximation error
||
G (ˆ θ ∗ ) − G π || D of GTD can be arbitrarily poor even if F Φ can representthe true value function arbitrarily well (but not exactly).
Such a disadvantage of GTD is mainly dueto the distribution mismatch in its objective function as we discuss in Section 3.3.
In the backward GVFs evaluation setting, GTD can perform even worse.
As we show in the followingexample, GTD may fail to learn the ground truth G π even if the function class F Φ is complete.
Notethat for such a case, GenTD converges to the ground truth as guaranteed by Theorem 2.
Example 1 (GTD Fails for Complete F Φ ) .
Consider a three-state Markov chain, with transition kernel P =
[[0 . 1 , 0 . 9 , 0] , [0 . 1 , 0 , 0 . 9] , [0 , 0 . 1 , 0 . 9]] (cid:62) , discount factor γ = 0 .
99 , and the reward function
R = [1 , 0 , 1] (cid:62) .
The back value function in this MDP is given by ¯ V = [8 . 1555 , 9 .
0389 , 9 . 0184] (cid:62) .
Suppose GTD is applied to solving the evaluation problem with the parameter space R θ = R .
Then, thereexists an off-policy distribution D such that using the perfect bases Φ = [8 . 1555 , 9 .
(cid:62) ,the optimal point ¯ θ ∗ learned by GTD still has non-zero approximation error, i.e., || Φ¯ θ ∗ − ¯ V || D ≥ 3 .
Experiments
We conduct empirical experiments to answer the following two questions: (a) can GenTD evaluateboth the forward and backward GVFs efﬁciently?
(2) how does GenTD compare with GTD interms of the convergence speed and the quality of the estimation results?
In our experiments, we
354 consider a variant of Baird’s counterexample [1, 45] with 7 states and 2 actions (see Figure 2 in
Appendix A).
We study the problem of evaluating two high-dimensional GVFs, the gradient of
Q-function: ∇ w Q π ∈ R 14 (forward GVF), and the gradient of logarithmic stationary distribution:∇
w log( µ π ) ∈
R 14 (backward GVF), associated with a soft-max policy parameterized by w ∈ R 14 .
We consider two types of feature matrices Φ for estimating the GVFs: complete feature (CFT) andincomplete feature (INCFT), where CFT has large enough expressive power so that the groundtrue GVF can be fully expressed by the function class F Φ , whereas INCFT does not have enoughexpressive power and cannot capture the ground true GVF exactly.
The discount factor γ is set tobe 0 .
99 in all tasks, and all curves in the plots are averaged over 20 independent runs.
The detailedexperimental setting is provided in Appendix A.
The learning curves for GenTD and GTD are provided in Figure 1.
We evaluate their performancesbased on the estimation error with respect to the ground truth GVF: (cid:107) Φ θ t − G π (cid:107) µ π .
Note that both∇ w Q π and ∇ w log µ π can be exactly computed in tabular setting, so that the estimator error of theground truth can be computed.
For the task of ∇ w Q evaluation, GenTD converges considerablyfaster and closer to the ground truth (i.e., smaller estimation error) than GTD.
which can be attributedto the larger conditional number λ G of GenTD.
For the task of ∇ w log µ π evaluation, GenTD movesfast towards the ground truth GVF, whereas GTD, although still converges, stays far away from theground truth GVF even with CFT, which matches with our Example 1.
As we discuss in Section 4,this is because GTD in the backward GVF evaluation setting has distribution mismatch in its objectivefunction, which can signiﬁcantly shift the optimal point from the ground truth GVF.
Conclusion
We studied the off-policy evaluation problem of both forward and backward GVFs.
We focusedon the class of GVFs with casual ﬁltering, which covers a wide range of multiple interrelated andpossibly high-dimensional GVFs.
We ﬁrst showed that GVFs in such a class is the ﬁxed point of ageneral Bellman operator.
Based on such a property, we proposed a new off-policy algorithm called
GenTD.
GenTD evaluates GVFs efﬁciently by jointly updating the GVF approximation parameter and a density ratio estimator, which adjusts the mismatch of the behavior policy and assists theconvergence to the ground truth GVFs.
We show that GenTD provably converges to the globally optimal point, and such an optimal point is guaranteed to converge to the ground truth GVFs aslong as the function expressive power is sufﬁciently large.
For future work, it is interesting to study nonlinear function approximation for GVFs evaluation
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[No]
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A] Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots
, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]