Beyond the Return: Off-policy Function Estimation under User-specified Error-measuring Distributions
Abstract
Off-policy evaluation often refers to two related tasks: estimating the expected return of a policy and estimating its value function (or other functions of interest, such as density ratios).
While recent works on marginalized importance sampling (MIS) show that the former can enjoy provable guarantees under realizable function approximation, the latter is only known to be feasible under much stronger assumptions such as prohibitively expressive discriminators.
In this work, we provide guarantees for off-policy function estimation under only realizability, by imposing proper regularization on the MIS objectives.
Compared to commonly used regularization in MIS, our regularizer is much more flexible and can account for an arbitrary user-specified distribution, under which the learned function will be close to the groundtruth.
We provide exact characterization of the optimal dual solution that needs to be realized by the discriminator class, which determines the datacoverage assumption in the case of value-function learning.
As another surprising observation, the regularizer can be altered to relax the data-coverage requirement, and completely eliminate it in the ideal case with strong side information.
Introduction
Off-policy evaluation (OPE) often refers to two related tasks in reinforcement learning (RL): estimating the expected return of a target policy using a dataset collected from a different behavior policy, versus estimating the policy’s value function (or other functions of interest, such as density ratios).
The former is crucial to hyperparameter tuning and verifying the performance of a policy before real-world deployment in offline RL [VLJY19; Pai+20; ZJ21].
The latter, on the other hand, plays an important role in (both online and offline) training, often as the subroutine of actor-critic-style algorithms [LP03; LSAB19], but is also generally more difficult than the former: if an accurate value function is available, one could easily estimate the return by plugging in the initial distribution.
Between the two tasks, the theoretical nature of off-policy return estimation is relatively well understood, especially in terms of the function-approximation assumptions needed for samplecomplexity guarantees.
Among the available algorithms, importance sampling (IS) and its variants [PSS00; TTG15; JL16] do not require any function approximation, but incur exponential-in-horizon variance; Fitted-Q Evaluation [EGW05; LVY19] can enjoy polynomial sample complexity under appropriate coverage assumptions, but the guarantee relies on the strong Bellman-completeness assumption on the function class; marginalized importance sampling (MIS) methods, which have gained significant attention recently [LLTZ18; XMW19; UHJ20; NCDL19], use two function classes to simultaneously approximate the value and the density-ratio (or weight) function and optimize minimax objectives.
Notably, it is the only family of methods known to produce accurate return estimates with a polynomial sample complexity, when the function classes only satisfy the relatively weak realizability assumptions (i.e., they contain the true value and weight functions).
In comparison, little is known about off-policy function estimation, and the guarantees are generally less desirable.
Not only do the limitations of IS and FQE on return estimation carry over to this more challenging task, but MIS also loses its major advantage over FQE: despite the somewhat misleading impression left by many prior works, that MIS can handle function estimation the same way as return estimation, 1 MIS for function estimation often requires unrealistic assumptions such as prohibitively expressive discriminators.
For concreteness, a typical guarantee for function estimation from MIS looks like the following (see e.g., Theorem 4 of [LLTZ18] and Lemmas 1 and 3 of [UHJ20]):
Proposition 1 (Function estimation guarantee for MIS, informal) .
Suppose the offline data distribution d D satisfies d D ( s, a ) > 0 , ∀ s, a .
Given value-function class Q with q π ∈ Q and weight class
[Equation]
for some appropriate population loss function L .
To enable the identification of the value function q π , the result requires the discriminator class W to be the space of all possible functions over the state-action space ( W = R S×A ).
In the finite-sample regime, using such a class incurs a sample complexity that depends on the size of the state-action space, which completely defeats the purpose of function approximation.
In addition, these results only hold asymptotically, where the function of interest can be exactly identified in a point-wise manner.
Such an overly strong guarantee is unrealistic in the finite-sample regime, where one can only hope to approximate the function well in an average sense under some distribution , i.e., finite-sample performance guarantees should ideally bound ∥ (cid:98) q − q π ∥ 2 ,ν for the learned (cid:98) q , where ∥ · ∥ 2 ,ν is ν -weighted 2-norm.
Such fine-grained analyses are non-existent in MIS.
Even in the broader literature, such results not only require Bellman-completeness-type assumptions [UIJKSX21], they also come with some fixed ν (which is not necessarily d D ; see Section 2) and the user has no freedom in choosing ν .
This creates a gap in the literature, as downstream learning algorithms that use off-policy function estimation as a subroutine often assume the estimation to be accurate under specific distributions [KL02; AYBBLSW19] (see details in Appendix A).
To summarize, below are two important open problems on off-policy function estimation:
Is it possible to obtain polynomial 3 sample complexity for off-policy function estimation, using function classes that only satisfy realizability-type assumptions?
Can we specify a distribution ν to the estimation algorithm, such that the learned function will be close to the groundtruth under ν ?
In this work, we answer both open questions in the positive.
By imposing proper regularization on the MIS objectives, we provide off-policy function estimation guarantees under only realizability assumptions on the function classes.
Compared to commonly used regularization in MIS [NCDL19;
ND20; YNDLS20], our regularizer is much more flexible and can account for an arbitrary userspecified distribution ν , under which the learned function will be close to the groundtruth.
We provide exact characterization of the optimal dual solution that needs to be realized by the discriminator, which determines the data-coverage assumption in value-function learning.
As another surprising observation, the regularizer can be altered to relax the data-coverage requirement, and in the ideal case completely eliminate it when strong side information is available.
Proof-of-concept experiments are also conducted to validate our theoretical predictions.
Related Works
Regularization in MIS The use of regularization is very common in the MIS literature, especially in DICE algorithms [NCDL19; NDKCLS19; YNDLS20].
However, most prior works that consider regularization use tabular derivations and seldom provide finite-sample function-approximation guarantees on even return estimation, let alone function estimation.
(An exception is the work of [UIJKSX21], who analyze related estimators under Bellman-completeness-type assumptions; see the next paragraph.)
More importantly, prior works provide very limited understanding in how choice of regularization affects learning guarantees, and have considered only naïve forms of regularization (state-action-
independent and typically under d D ), under which different forms of regularization are essentially treated equally under a coarse-grained theory [YNDLS20].
In contrast, we provide much more fine-grained characterization of the effects of regularization, which leads to novel insights about how to design better regularizers, and existing DICE estimators are subsumed as special cases of our method when we choose very simple regularizers (see Remark 3 in Section 5).
Fitted-Q Evaluation (FQE) Outside the MIS literature
, one can obtain return and value-function estimation guarantees via FQE [DJW20; CJ19; LVY19; UIJKSX21].
However, it is well understood that FQE and related approaches require Bellman-completeness-type assumptions, such as the function class being closed under the Bellman operator.
Even putting aside the difference between completeness vs. realizability, we allow for a user-specified error-measuring distribution, which is not available in FQE or any other existing method.
The only distribution these methods are aware of is the data distribution d D , and even so, FQE and variants rarely provide guarantees on ∥ (cid:98) q − q π ∥ 2 ,d D , but often on the Bellman error (e.g., ∥ (cid:98) q − T π (cid:98) q ∥ 2 ,d D ) instead [UIJKSX21], and obtaining guarantees on a distribution of interest often requires multiple indirect translations and loose relaxations.
LSTDQ
Our analyses focus on general function approximation.
When restricted to linear classes, function estimation guarantees for q π under d D can be obtained by LSTDQ methods [LP03; BY09;
DNP14] when the function class only satisfies realizability of q π [PKBK22].
However, this requires an additional matrix invertibility condition (see Assumption 3 of [PKBK22]), and it is still unclear what this condition corresponds to in general function approximation.
Moreover, many general methods—including MIS [UHJ20] and other minimax methods [ASM08; XCJMA21]—coincide with LSTDQ in the linear case, so the aforementioned results can be viewed as a specialized analysis leveraging the properties of linear classes.
PRO-RL [ZHHJL22]
Our key proof techniques are adapted from [ZHHJL22], whose goal is offline policy learning.
They learn the importance weight function w π for a near-optimal π , and provide ∥ (cid:98) w − w π ∥ 2 ,d D guarantees as an intermediate result.
Despite using similar technical tools, our most interesting and surprising results are in the value-function estimation setting, which is not considered by [ZHHJL22].
Our novel algorithmic insights, such as incorporating error-measuring distributions and approximate models in the regularizers, are also potentially useful in [ZHHJL22]’s policy learning setting.
Our analyses also reveal a number of important differences between OPE and offline policy learning, which will be discussed in Appendix A.
Preliminaries
We consider off-policy evaluation (OPE) in Markov Decision Processes (MDPs).
An MDP is specified by its state space S , action space A , transition dynamics P : S ×
A → ∆( S ) ( ∆( · ) is the probability simplex), reward function R : S ×
A → ∆([0 , 1]) , discount factor γ ∈ [0 , 1) , and an initial state distribution µ 0 ∈ ∆( S ) .
We assume S and A are finite and discrete, but their cardinalities can be arbitrarily large.
Given a target policy π : S → ∆( A ) , a random trajectory s 0 , a 0 , r 0 , s 1 , a 1 , r 1 , . . .
can be generated as s 0 ∼ µ 0 , a t ∼ π ( ·| s t ) , r t ∼ R ( ·| s t , a t ) , s t +1 ∼ P ( ·| s t , a t ) , ∀ t ≥ 0 ; we use E π and P π to refer to expectation and probability under such a distribution.
The expected discounted return (or simply return) of π is J ( π ) :=
E π [ (cid:80) t γ t r t ] .
The Q-value function of π is the unique solution of the Bellman equations q π = T
π q π , with the Bellman operator T π : R S×A → R S×A defined as ∀ q ∈ R S×A , ( T π q )( s, a ) :
= E r ∼ R ( ·| s,a )
[ r ] + γ ( P π q )( s, a ) .
Here P π ∈ R |S×A|×|S×A| is the stateaction transition operator of π , defined as ( P π q )( s, a ) :
=
E s ′ ∼ P ( ·| s,a ) ,a ′ ∼ π ( ·| s ′ )
[ q ( s ′ , a ′ )] .
Functions over S × A (such as q ) are also treated as |S × A| -dimensional vectors interchangeably.
In OPE, we want to estimate q π and other functions of interest based on a historical dataset collected by a possibly different policy.
As a standard simplification, we assume that the offline dataset consisting of n i.i.d.
tuples { ( s i , a i , r i , s ′ i ) } ni =1 sampled as ( s i , a i ) ∼ d D , r ∼ R ( ·| s i , a i ) , and s ′ i ∼ P ( ·| s i , a i ) .
We call d D the (offline) data distribution.
As another function of interest, the (marginalized importance) weight function w π is defined as w π ( s, a ) :
=
d π ( s, a ) /d D ( s, a ) , where d π ( s, a ) =
(1 − γ ) (cid:80) ∞
t =
γ t P π [ s t = s, a t = a ] is the discounted state-action occupancy of π .
For technical convenience we assume d D ( s, a ) > 0 ∀ s, a , so that quantities like w π are always well defined and finite.
5 Similarly to q π , w π also satisfies a recursive equation, inherited from the Bellman flow equation for d π : d π = (1 − γ )
µ π 0 + γ (cid:101) P π d π , where ( s, a ) ∼ µ π 0 ⇔ s ∼ µ 0 , a ∼ π ( ·| s ) is the initial state-action distribution, and (cid:101) P
π
:= ( P π ) ⊤ is the transpose of the transition matrix.
Function Approximation We will use function classes Q and W to approximate q π and w π , respectively.
We assume finite Q and W , and extension to infinite classes under appropriate complexity measures (e.g., covering number) is provided in Appendix G.
Additional Notation ∥·∥ 2 ,ν := (cid:112) E ν [( · ) 2 ] is the weighted 2-norm of a function under distribution ν .
We also use a standard shorthand f
( s, π ) :
= E a ∼ π ( ·| s )
[ q ( s, a )] .
Elementwise multiplication between two vectors u and v of the same dimension is u ◦ v , and elementwise division is u/v .
Value-function Estimation
In this section we show how to estimate (cid:98) q ≈ q π with guarantees on ∥ (cid:98) q − q π ∥ 2 ,ν for a user-specified ν , and identify the assumptions under which provable sample-complexity guarantees can be obtained.
We begin with the familiar Bellman equations, that q π is the unique solution to:
[Equation]
While the above set of equations uniquely determines q = q π , this is only true if we can enforce all the |S × A| constraints, which is intractable in large state-space problems.
In fact, even estimating (a candidate q ’s violation of) a single constraint is infeasible as that requires sampling from the same state multiple times, which is related to the infamous double-sampling problem [Bai95].
To overcome this challenge, prior MIS works often relax Eq.(1) by taking a weighted combination of these equations, e.g.,
[Equation]
Instead of enforcing |S × A| equations, we only enforce their linear combinations; the linear coefficients are d D ( s, a ) · w ( s, a ) , and w belongs to a class W with limited statistical capacity to enable sample-efficient estimation.
While each constraint in Eq.(2) can now be efficiently checked on data, this comes with a big cost that a solution to Eq.(2) is not necessarily q π .
Prior works handle this dilemma by aiming lower: instead of learning (cid:98) q ≈ q π , they only learn (cid:98) q that can approximate the policy’s return, i.e. E s ∼ µ
[ (cid:98) q
( s, π )]
≈ J ( π ) =
E s ∼ µ
[ q π ( s, π )] .
While [UHJ20] show that the latter is possible when w π ∈ W , they also show explicit counterexamples where (cid:98) q ̸ = q π even with infinite data.
As a result, how to estimate (cid:98) q ≈ q π under comparable realizability assumptions (instead of the prohibitive W = R S×A as in Proposition 1) is still an open problem.
4.1 Estimator
We now describe our approach to solving this problem.
Recall that the goal is to obtain error bounds for ∥ (cid:98) q − q π ∥ 2 ,ν for some distribution ν ∈ ∆( S × A ) specified by the user.
Note that we do not require information about r and s ′ that are generated after ( s, a ) ∼ ν
and only care about the ( s, a ) marginal itself, so the user can pick ν in an arbitrary manner without knowing the transition and the reward functions of the MDP.
We assume that ν is given in a way that we can take its expectation E ( s,a ) ∼ ν
[( · )] , and extension to the case where ν is given via samples is straightforward.
To achieve this goal, we first turn Eq.(1) into an equivalent constrained convex program : given a collection of strongly convex and differentiable functions { f s,a : R → R } s,a —we will refer to the collection as f and discuss its choice later—consider
[Equation]
The constraints here are the same as Eq.(1).
Since Eq.(1) uniquely determines q = q π , the feasible space of Eq.(3) is a singleton , so we can impose any objective function on top of these constraints (here we use E ( s,a ) ∼ ν [ f s,a ( q ( s, a ))] ) and it will not change the optimal solution (which is always q π , the only feasible point).
As we will see, however, f = { f s,a : R → R } s,a will serve as an important regularizer in the function-approximation setting and is crucial to our estimation guarantees.
Remark 1 ( ( s, a ) -dependence of f ) .
Regularizers in prior works are ( s, a ) - independent [NCDL19;
YNDLS20; ZHHJL22].
As we will see in Section 4.3, allowing for ( s, a ) -dependence is very important for designing regularizers with improved guarantees and performances.
We now rewrite (3) in its Lagrangian form, with d D ◦ w serving the role of dual variables:
[Equation]
Finally, our actual estimator approximates Eq.(4) via finite-sample approximation of the population loss L qf , and searches over restricted function classes Q and W for q and w , respectively:
[Equation]
where (cid:98) L qf ( q, w ) =
E ν [ f s,a ( q ( s, a ))] + 1 n (cid:80) ni
=1 w ( s i , a i ) ( r i + γq ( s ′
i , π ) − q ( s i , a i )) .
Intuition for identification Before giving the detailed finite-sample analysis, we provide some high-level intuitions for why we can obtain the desired guarantee on ∥ (cid:98) q − q π ∥ 2 ,ν .
Note that Eq.(4) is structurally similar to Eq.(2), and we still cannot verify the Bellman equation for q π in a per-stateaction manner, so the caveat of Eq.(2) seems to remain; why can we identify q π under ν ?
The key here is to show that it suffices to check the loss function L qf only under a special choice of w (as opposed to all of R S×A ).
Importantly, this special w is not w = w π ; 6 rather, it is the saddle point of our regularized objective L qf : let ( q π , w ∗ f ) be a saddle point of L qf (we will give the closed form of w ∗ f later).
As long as w ∗ f ∈ W —even if W is extremely “simple” and contains nothing but w ∗ f —we can identify q π .
To see that, it is instructive to consider the special case of W = { w ∗ f } and the limit of infinite data.
In this case, our estimator becomes arg min q ∈Q L qf ( q, w ∗ f ) .
By the definition of saddle point:
[Equation]
While this shows that q π is a minimizer of the loss, it does not imply that it is a unique minimizer.
However, identification immediately follows from the convexity brought by regularization: since f : R → R is strongly convex, q → E ν [ f s,a ( q ( s, a ))] as a mapping from R S×A to R is strongly convex under ∥ · ∥ 2 ,ν (see Lemma 7 in Appendix B for a formal statement and proof), and L qf ( q, w ∗ f ) inherits such convexity since the other terms are affine in q .
It is then obvious that q π is the unique minimizer of L qf ( q π , w ∗ f ) up to ∥ · ∥ 2 ,ν , that is, any minimizer of L qf must agree with q π on ( s, a ) pairs supported on ν .
Our finite-sample analysis below shows that the above reasoning is robust to finite-sample errors and the inclusion of functions other than w ∗ f in W .
4.2 Finite-sample Guarantees
In this subsection we state the formal guarantee of our estimator for q π and the assumptions under which the guarantee holds.
We start with the condition on the regularization function f :
Assumption 1 (Strong convexity of f ) .
Assume f s,a : R → R is nonnegative, differentiable, and
M q -strongly convex for each s ∈ S , a ∈ A .
In addition, assume both f s,a and its derivative f ′ s,a take finite values for any finite input.
This assumption can be concretely satisfied by a simple choice of f s,a ( x ) = 12 x 2 , which is independent of ( s, a ) and yields M q = 1 .
Alternative choices of f will be discussed in Section 4.3.
Next are the realizability and boundedness of W and Q :
Assumption 2 (Realizability) .
Suppose w ∗ f ∈ W , q π ∈ Q .
Assumption 3 (Boundedness of W and Q ) .
Suppose W and Q are bounded, that is,
[Equation] ∞ .
As a remark, Assumption 2 implicitly assumes the existence of w ∗ f .
As we will see in Section 4.3, the existence and finiteness of w ∗ f is automatically guaranteed given the finiteness of f ′ s,a (Assumption 1) and d D ( s, a ) > 0 ∀ s, a .
More importantly, Assumptions 2 and 3 together imply that ∥ q π ∥ ∞ ≤ C q Q and ∥ w ∗ f ∥ ∞ ≤ C q W , which puts constraints on how small C q Q and C q W can be.
For example, it is common to assume that C q Q = 11 − γ , i.e., the maximum possible return when rewards are bounded in [0 , 1] , and this way ∥ q π ∥ ∞ ≤ C q Q will hold automatically.
The magnitude of ∥ w ∗ f ∥ ∞ and C q W , however, is more nuanced and interesting, and we defer the discussion to Section 4.3.
Now we are ready to state the main guarantee for identifying q π .
All proofs of this section can be found in Appendix B.
Theorem 2.
Suppose Assumptions 1, 2, 3 hold.
Then, with probability at least 1 − δ ,
[Equation]
where ϵ
[Equation]
Theorem 2 shows the desired bound on ∥ (cid:98) q − q π ∥ 2 ,ν , which depends on the magnitude of functions in W and Q as well as their logarithmic cardinalities, which are standard measures of statistical complexity for finite classes.
One notable weakness is the O ( n − 1 / 4 ) slow rate; this is due to translating the ϵ qstat =
O ( n − 1 / 2 ) deviation between L and (cid:98) L into ∥ (cid:98) q − q π ∥ 2 ,ν via a convexity argument, which takes square root of the error.
The possibility of and obstacles to obtaining an O ( n − 1 / 2 ) rate will be discussed in Section 7.
4.3 On the Closed Form of w ∗ f and the Data Coverage Assumptions
One unusual aspect of our guarantees in Section 4.2 is that we do not make any explicit data coverage assumptions, yet such assumptions are known to be necessary even for return estimation (typically the boundedness of w π =
d π /d D ).
Indeed, our data-coverage assumption is implicit in Assumptionsand 3, which require ∥ w ∗ f ∥ ∞ ≤ C q W < ∞ .
If data fails to provide sufficient coverage, ∥ w ∗ f ∥ ∞ will be large and our bound in Theorem 2 will suffer due to a large value of C q W .
To make the data coverage assumption explicit, we provide the closed-form expression of w ∗ f :
Lemma 3.
The saddle point of (4) is ( q π , w ∗ f ) =
arg min q arg max w L qf ( q, w ) , where
[Equation]
Here f ′ ( q π ) is the shorthand for [ f ′ s,a ( q π ( s, a ))] s,a ∈ R S×A .
The closed-form expression in Eq.(6) looks very much like a density ratio: if we replace ν ◦ f ′ ( q π ) with µ π 0 , we have ( I − γ (cid:101) P π ) − 1 µ π 0 =
d π / (1 − γ ) , and the expression would be the ratio between d π and d D (up to a horizon factor).
Therefore, w ∗ f can be viewed as the density ratio of π against d D when π starts from the “fake” initial distribution ν ◦ f ′ ( q π ) .
However, ν ◦ f ′ ( q π ) is in general not a valid distribution, as it is not necessarily normalized or even non-negative, making ∥ w ∗ f ∥ ∞ difficult to intuit.
Below we give relaxations of ∥ w ∗ f ∥ ∞ , which are more interpretable and give novel insights into how to relax the data-coverage assumption via tweaking f .
Proposition 4.
[Equation] w [Equation] ∥ ∞ · ∥ f ′ ( q π ) ∥ ∞ , where d πν is the discounted state-action occupancy of π under ν as the initial state-action distribution.
The proposition states that ∥ w ∗ f ∥ ∞ can be bounded if data provides sufficient coverage over d πν , and if f ′ ( q π ) is bounded.
The former shows that d D needs to cover not only ν , but also state-action pairs reachable by π starting from ν .
The latter is easily satisfied, and can be bounded again for concrete choices of f , e.g. ∥ f ′ ( q π ) ∥ ∞
≤ ∥ q π ∥ ∞ ≤ 11 − γ for f s,a ( x ) = 12 x 2 .
Designing f to relax the coverage assumption Lemma 3 shows that the coverage assumption (bounded ∥ w ∗ f ∥ ∞ ) depends on f (or rather its derivative f ′ ), which opens up the possibility of properly designing f to relax it.
In fact, we could completely eliminate the coverage assumption if we could set f ′ ( q π ) = 0 , but that would require unrealistically strong side information.
As a concrete example, consider f s,a ( x ) = 12 ( x − q π
( s, a )) 2 , and it is easy to verify that f ′ s,a ( q π ( s, a )) =
x − q π ( s, a ) | x = q π ( s,a ) = 0 .
Compared to f s,a ( x ) = 12 x 2 , the new f essentially adds a 1st-order term q π ( s, a ) · x to change w ∗ f , while leaving the convexity required by
Assumption 1 intact, which only depends on the 2nd-order term 12 x 2 .
Of course, this is not a viable choice of f in practice as it requires knowledge of q π , which is precisely our learning target.
While the reason f s,a ( x ) = 12 ( x − q π ( s, a )) 2 can eliminate the coverage requirement is obvious retrospectively ( q π already minimizes E ν [ f ( q )] even without any data), our analyses apply much more generally and characterize the effects of arbitrary f on the coverage assumption.
Inspired by this example, we can consider practically feasible choices such as f s,a ( x ) = 12 ( x − (cid:101) q
( s, a ))
2 , where (cid:101) q is an approximation of q π obtained by other means, e.g. a guess based on domain knowledge.
If (cid:101) q ≈ q π , our estimator enjoys significantly relaxed coverage requirements.
But even if (cid:101) q is a poor approximation of q π , it does not affect our estimation guarantees as long as the condition implied by Proposition 4 is satisfied.
(In fact, f s,a ( x ) = 12 x 2 is a special case of (cid:101) q ≡ 0 .)
Such a use of approximate models is similar to how doubly robust estimators [DLL11; JL16; TB16] enjoy reduced variance given an accurate model, and remain unbiased even if the approximate model is arbitrarily poor.
We will show in Section 6 that this idea is empirically effective.
Weight-function Estimation
Similar to value-function estimation, our methodology can also be applied to estimate the weight function w π .
Due to the similarity with Section 4 in the high-level spirit, we will be concise in this section and only explain in detail when there is a conceptual difference from Section 4.
Some notations (such as the function classes W and Q ) will be abused, but we emphasize that this section considers a different learning task than Section 4, so they should be viewed as different objects (e.g., the realizability assumptions for W and Q below will be different from those in Section 4).
As before, we assume that the user provides a distribution 7 η ∈ ∆( S × A ) and our goal is to develop an estimator with guarantees on ∥ (cid:98) w − w π ∥ 2 ,η .
Analogous to Section 4, consider min w E ( s,a ) ∼ η [
[Equation]
Here f = { f s,a } s,a will need to satisfy similar assumptions as in Section 4.
The constraints are the Bellman flow equations with a change of variable d ( s, a ) =
d D ( s, a ) · w ( s, a ) .
Their unique solution is d ( s, a ) =
d π ( s, a ) (and hence w ( s, a ) =
d π ( s, a ) /d D
( s, a ) ), thus the feasible space is again a singleton, and the objective does not alter the optimal solution.
We then use dual variables q to rewrite (7) in its Lagrangian form: min w max q L wf ( q, w ) :=
[Equation]
We approximate the saddle-point solutions by optimizing the empirical loss (cid:98) L wf over restricted function classes W , Q : (cid:98) w = [Equation]
max [Equation] ∈Q (cid:98) L wf ( q, w ) , where (cid:98) L wf ( q, w ) :=
E η [ f s,a ( w ( s, a ))]
+ {+(1 − γ ) 
+}
− γ n 0 (cid:80) n 0 j =1 q ( s j , π ) + 1 n (cid:80) ni
=1 w ( s i , a i ) ( γq ( s ′ i , π ) − q ( s i , a i )) , and { s j } n 0 j =1 is a separate dataset sampled i.i.d.
from µ 0 to provide information about the initial distribution.
We provide the closed-form expression for the saddle point of L wf below, which resembles the
Q-function for a proxy reward function f ′ ( w π ) ◦ η/d D .
Lemma 5.
The closed form solutions of (8) are ( w π , q ∗ f ) =
arg min w arg max q L wf ( q, w ) , where
[Equation]
Remark 2 (Data Coverage Assumption) .
As we will see, the only data coverage assumption we need is the boundedness of w π = d π /d D .
Since w π is the function of interest and practical algorithms can only output functions of well-bounded ranges, such an assumption is an essential part of the learning task itself and hardly an additional requirement.
Moreover, unlike Section 4, changing f here will not affect the data-coverage assumption, though it still alters q ∗ f , and a properly chosen f (e.g., with f ′ ( w π ) ≈ 0 ) can still result in a q ∗ f with small magnitude and thus make learning easier.
Remark 3 (Connection to DualDICE) .
We can recover DualDICE [NCDL19] by choosing f s,a ( x ) = 12 x 2 and ν = d D .
Despite producing the same estimator, the derivations and assumptions under which the two works analyze the estimator are different.
Their Theorem 2 only provides return estimation guarantees, and depends on an implicit assumption of highly expressive function classessimilar to Proposition 1.
Moreover, they do not characterize how the choice of f can affect the learning guarantees (their f is ( s, a ) -independent).
This is one of the main insights of our paper and leads to the discovery of more practical regularizers, e.g. f s,a ( x ) = 12 ( x − ˜ w ( s, a )) 2 with model (cid:101) w .
Below we present the assumptions, then learning guarantee for (cid:98) w .
Assumption 4 (Strongly Convex Objective) .
Suppose for all s, a , f s,a is differentiable, non-negative, and M w -strongly convex.
Further, suppose f s,a and its derivative take finite values on any finite inputs, and let C wf := max w ∈W || f ( w ) || ∞ .
Assumption 5 (Realizability) .
Suppose w π ∈ W , q ∗ f ∈ Q .
Assumption 6 (Bounded W and Q ) .
Let C w W := max w ∈W || w || ∞ and C w Q :=
max q ∈Q || q || ∞ .
Suppose W and Q are bounded function classes, that is, C w W < ∞ and C w Q < ∞ .
Theorem 6.
Suppose Assumptions 4, 5, 6, hold.
Then w.p. ≥ 1 − δ , ∥ (cid:98) w − w π ∥ 2 ,
η ≤ 2 (cid:113) ϵ wstat M w , where
[Equation]
Experiments
We now provide experimental results to verify our theoretical predictions and insights.
As [YNDLS20] have performed extensive experiments on return estimation with simple regularization ( f s,a ( x ) = 12 x 2 ), we focus on the task of q π estimation, and the following two questions unique to our work:
Q1.
When the goal is to minimize ∥ (cid:98) q − q π ∥ 2 ,ν , how much benefit does regularizing with ν bring in practice, compared to regularizing with other distributions (or no regularization at all)?
Q2.
Can incorporating (even relatively poor) models in regularization (e.g., f s,a ( x ) = 12 ( x − (cid:101) q ( s, a ))from Section 4.3) improve estimation?
Setup We study these questions in a large tabular Gridwalk environment [NCDL19; YNDLS20], with a deterministic target policy π that is optimal, and a behavior policy that provides limited coverage over the target policy; see Appendix D for further details.
To mimic the identification challenges associated with restricted function classes, we use a linear function class Q = { Φ ⊤ α : α ∈ R d } and discriminator class W = { (cid:101)
Φ ⊤ β : β ∈ R k } , where k < d ≪ |S × A| .
The features Φ ∈ R |S×A|× d , (cid:101)
Φ ∈ R
|S×A|× k are chosen to satisfy the realizability assumptions of all estimators.
Under linear classes, our estimator (Eq.(5)) becomes a convex optimization problem with d variables and k linear constraints, and can be solved by standard packages.
This allows us to avoid difficult minimax optimization—which is still an open problem in the MIS literature—and focus on the statistical behaviors of our estimators, which is what our theoretical predictions are about.
Remark 4.
When no regularization is used, our linear estimator coincides with MQL [UHJ20].
If we further had (cid:101) Φ =
Φ , the estimator would coincide with LSTDQ.
While Section 2 mentioned that LSTDQ enjoys function-estimation guarantees [PKBK22] (and folklore suggests they extend to (cid:101) Φ ̸ = Φ ), the guarantee only holds in the regime of k ≥ d , i.e., the k linear constraints are overdetermined .
In our case, however, we have under-determined constraints ( k < d ), creating a more challenging learning task (which our theory can handle) where LSTDQ’s guarantees do not apply.
Choice of Distributions We consider a set of diverse distributions V = { d D , µ π 0 , d π , U, p } , where
U is uniform over S × A and p ∝ ( d π ◦
I [ w π > 50]) .
The distribution p isolates the least-covered states reached by π , which makes learning an accurate Q-function on ν a harder task.
Results for Q
We use a default regularizer f = 12 x 2 with different regularizing distributions ν ∈ V , and measure ∥ (cid:98) q − q π ∥ 2 ,ν ′ for different ν ′ .
The results are shown in Figure 1, which exhibit the expected trend: for example, regularizing with ν = p performs poorly when the error is measured under ν ′ =
d D and U due to the large mismatch between ν and ν ′ .
However, when ν ′ =
p (rightmost panel), regularizing with ν = p significantly outperforms others.
Similar behaviors can also be observed on U , though they are certainly not absolute
(e.g., ν =
d D does not do very well on
[Figure] [Figure]
[Figure]
log | J ( ) J ( )| -0.
log q q 2, d D -0.2 0.
log q q 2, U -0.5 -0.
log q q 2, p none d D U d p
[Figure]
ν ′ =
d D ), which suggests potential directions for more refined and accurate theory.
Moreover, using no regularization (“none”) generally does not perform well for any ν ′ , but still manages to achieve a high accuracy for return estimation J ( π ) , which is consistent with prior theory [e.g., UHJ20] that return estimation does not require regularization.
Results for Q
We now use f s,a ( x ) = 12 ( x − (cid:101) q ( s, a )) 2 with different (cid:101) q to verify how the quality of (cid:101) q affect estimation accuracy.
We first consider a “uniform model” (cid:101) q = mq π
+ (1 − m ) q , where q is a constant and m ∈ [0 , 1] controls the quality (cid:101) q .
As shown from Panels 1 & 3 in Figure 2, our estimator’s accuracy generally improves with a better (cid:101) q (i.e., as m increases).
Moreover, equipping (cid:101) q with an appropriate regularizing distribution ν (e.g., ν = U for both panels) can significantly outperform no regularization, even with a very poor (cid:101) q
(e.g., m = 0 . 1 ).
It also outperforms the model prediction itself (i.e., (cid:98) q
= (cid:101) q ), showing that the improvement is not due to our estimator simply taking predictions from (cid:98) q , but using the regularization to better assist the identification of q π from data.
The previous model’s quality is uniform across S × A .
We then consider a scenario where (cid:101) q is zeroed out outside p ’s support, making it only a good approximation of q π on p .
In this case, we see that regularization cannot benefit much from the model when the error is measured on ν ′ =
U (Panel 2), but when ν ′ =
ν = p (Panel 4), regularization can still bring benefits, as expected from our theory.
Discussion and Conclusion
In this paper we showed that proper regularization can yield function-estimation guarantees for MIS methods under only realizable function approximation.
Compared to prior works, our regularizer is more flexible and can accommodate a user-specified error-measuring distribution.
Further theoretical investigation provides fine-grained characterization of how the choice of regularization affects learning guarantees, which leads to the discovery of regularizers that incorporate approximate models (such as (cid:101) q ).
While the superiority of such regularizers is perhaps obvious retrospectively, it is not allowed in the prior works’ derivation that assumes ( s, a ) -independent regularization, and our theoretical results provide a deep understanding for even more general regularization schemes.
In Appendix A, we provide further discussions on two topics: (1) the barriers to obtaining a faster O ( n − 1 / 2 ) rate, and (2) comparison to [ZHHJL22] reveals interesting differences between off-policy function estimation and policy learning, and insights in this paper may also be useful for the policy learning task.
Acknowledgments and Disclosure of Funding
The authors thank Jinglin Chen, Wenhao Zhan and Jason Lee for valuable discussions during the early phase of the project, and Anonymous Reviewer e62P for insightful feedback that helped improve this paper during the review process.
NJ acknowledges funding support from ARL Cooperative
Agreement W911NF-17-2-0196, NSF IIS-2112471, NSF CAREER IIS-2141781, and Adobe Data
Science Research Award.
A Further Discussions
Motivating Examples Below, we give a thorough discussion, through four examples of how offpolicy function estimation used in downstream learning algorithms.
We highlight the discrepancies between what these algorithms assume about the function estimates and what existing work is able to achieve, and demonstrate how our work closes these gaps.
Batch Learning: [LSAB19] design an off-policy policy gradient algorithm that requires estimating the density-ratio w π to correct the offline data distribution to the on-policy distribution.
In their convergence analysis, they assume access to a blackbox w π estimator that is accurate under d D , and refer to [LSAB19] as a possible method.
However, as per Proposition 1, [LLTZ18] and existing works do not provide desirable guarantees for such a task.
Online Reinforcement Learning: The seminal paper of [KL02] designs the CPI algorithm for onpolicy policy improvement, which inspired popular empirical algorithms such as TRPO and PPO.
CPI requires an oracle for estimating the advantage function ( ≈ value function up to offset) accurately under the on-policy distribution, i.e., distribution induced by the current policy (see their Sec 7.1).
While this is easy to do by simple squared-loss regression onto on-policy trajectories, it can be sample-inefficient as it fails to leverage off-policy data collected by previous policies.
On the other hand running something like TD on all data considers a distribution different from the on-policy one.
Our method offers a direct solution: use all data in the Bellman error part of the objective, and only use on-policy trajectories in the regularizer.
Online Reinforcement Learning: [AYBBLSW19] designs a no-regret policy optimization algorithm assuming access to value-function estimation oracles.
In their Theorem 5.1, they assume that the oracle outputs an estimate of q π that is accurate under ν = d π ∗ .
While d π ∗ is obviously not accessible to us and our method does not apply as-is, one might use our theoretical insights to design heuristics, such as up-weighting high-reward states in the offline distribution, as a way to mimic d π ∗ .
Model selection in Offline Return Estimation: Model selection in offline return estimation:
Hyperparameter tuning is a huge practical hurdle in offline return estimation [Pai+20], i.e., all OPE estimators for return estimation (except for importance sampling which has exponential variance) require some form of function approximation, and it is hard to choose the right function class with offline data alone.
To address this issue, [ZJ21] proposes a model selection process over candidate function estimates of q π , which must be provided by base algorithms that perform function estimation.
Function Estimation & Downstream Tasks Online algorithms using off-policy function estimation as a subroutine, such as [KL02; AYBBLSW19], may require the estimates to be accurate on unknown distributions such as d π or d π ∗ (where π ∗ is the optimal policy), which may not be immediately accessible to the user.
The user may be able to use domain knowledge to “guess" a distribution ν close to or covering the unknown distribution of interest.
Then our guarantees for function estimation over ν could similarly, with a change of distribution, be converted to guarantees on the true distribution of interest.
To this end, an important avenue of future work involves a thorough investigation of how our off-policy function estimation method interacts with such downstream learning algorithms, their assumptions, and their guarantees, as well as how our method can tailored to improve downstream tasks.
Faster rate One weakness of our result is the O ( n − 1 / 4 ) slow rate of estimation.
While O ( n − 1 / 2 ) generalization error bounds for related stochastic saddle point exist [ZHWZ21], they only apply to strongly-convex-strongly-concave problems, whereas our problem is strongly-convex-non-stronglyconcave ( L qf is affine in w and L wf is affine in q ), making the result not directly applicable.
One immediate idea is to introduce dual regularization to make our objectives also strongly concave in the discriminator.
However, while primal regularization does not change the feasible space and guarantees that the learned function will be q π (or w π , respectively), dual regularization does change the optimal solution, introducing a bias.
This leads to a trade-off between the improvement in error bounds due to strong concavity and the additional bias, and our preliminary investigation shows that an optimal trade-off between the two sources of errors still leads to an O ( n − 1 / 4 ) rate.
Therefore, improving the rate (if it is possible at all) will require novel technical tools for the generalization analyses of strongly-convex-non-strongly-concave stochastic saddle point problems, which will be an interesting future direction.
On a related note, while the rate for estimating q π and w π is only O ( n − 1 / 4 ) , we can combine them in a doubly robust form to get O ( n − 1 / 2 ) rate for return estimation by careful choices of the regularizing distributions ν and η ; see Appendix F for details.
Comparison to off-policy learning As mentioned earlier, our results are enabled by technical tools adapted from [ZHHJL22], whose work focuses on off-policy policy learning and learns w π for a near-optimal π that is accurate under d D as an intermediate step.
While most of our surprising observations are in the value-function learning scenario (Section 4), comparing our guarantee for learning w π (Section 5) to that of [ZHHJL22] still yields interesting observations about the difference between off-policy evaluation and learning.
Most notably, we do not need to control the strength of regularization in Eq.(3), since the feasible space is a singleton and there is no objective before we introduce E ν [ f ( q )] .
In contrast, the feasible space is not a singleton in [ZHHJL22] (it is the space of all possible occupancies) and there is already a return optimization objective, so [ZHHJL22] need to carefully control the strength of their regularization.
As a consequence, [ZHHJL22] obtain
O ( n − 1 / 6 ) rate, showing how off-policy learning is potentially more difficult than off-policy function estimation.
Another interesting difference is related to our exact characterization of w ∗ f and q ∗ f : [ZHHJL22] do not have a closed-form expression for their optimal dual solution.
Such a lack of direct characterization leads to requiring additional assumptions to guarantee the boundedness of such variables (see their Assumptions 11 and 12), which is not a problem in our setting.
Finally, our analyses lead to novel algorithmic ideas such as using state-action-dependent regularizers and incorporating approximate models in the regularizers, which are potentially also useful for policy learning.
B Proofs for SectionB.1 Proof of Theorem
From Assumption 1 and Lemma 7, we know that the regularization function E ν [ f s,a ( q ( s, a ))] is an
M -strongly convex function in q on the ∥ · ∥ 2 ,ν norm.
Now consider L qf ( q, w ∗ f ) , the Lagrangian function (4) at the optimal discriminator w ∗ f .
Since L qf ( q, w ∗ f ) is composed of the regularization function plus terms that are linear in q , L qf ( q, w ∗ f ) is also an M -strongly convex function in q .
As ( q π , w ∗ f ) is the saddle point solution of L qf , we know q π = arg min q L qf ( q, w ∗ f ) .
Then from the strong convexity of L qf ,
[Equation]
where ϵ qstat is given in Lemma 8.
We provide the helper lemmas and their proofs below:
Lemma 7.
Suppose f s,a : R → R is M -strongly convex.
Then E ν [ f s,a ( q ( s, a ))] : R | SA | → R is
M -strongly convex on ∥ · ∥ ν .
Proof.
From the strong convexity of f s,a , for any x, y ∈ R ,
[Equation]
Then for q, q ′ ∈ R | SA | ,
[Equation]
since ∇ q E ν [ f s,a ( q ( s, a ))]
=
ν ◦ f ′ s,a ( q ) , which gives our result.
Lemma 8.
Suppose Assumption 3 holds.
Then for all ( q, w ) ∈ Q × W , w.p. ≥ [Equation]
[Equation]
where ϵ qstat =
[Equation]
C
[Equation]
+
[Equation]
+ γ
[Equation]
C
[Equation]
W C [Equation] Q
[Equation]
2log 2 |W||Q| δ n .
Proof.
From the linearity of the expectation, it is clear that L qf ( q, w ) =
E [ (cid:98) L qf ] .
Let l i =
w [Equation] s i , a i
[Equation] i [Equation]
+ γq [Equation] s
[Equation] i [Equation] π
[Equation] − q ( s i , a i )) . From Assumption 3,
[Equation]
Then using Hoeffding’s inequality with union bound, for all q, w ∈ Q × W , w.p. ≥ 1 − δ , (cid:12) [Equation] n [Equation]
i =
[Equation]
Lemma 9.
Under Assumptions 1, 2, 3, w.p. ≥ 1 − δ ,
[Equation]
where ϵ qstat is given in Lemma 8.
Proof.
Let (cid:98) w
[Equation]
q ) :
= arg max w ∈W (cid:98) L qf ( q, w ) .
We decompose the error as follows:
[Equation]
Combining the terms gives the result, and we provide a brief justification for each inequality below.
Terms (2) and (5) follow from Lemma 8.
Term (1) ≥ 0 since ( q π , w ∗ f ) is the saddlepoint solution.
Term (3) ≥ 0 , since (cid:98) q = arg min q ∈Q (cid:98)
L qf ( q, (cid:98) w ( q )) , and q π ∈ Q .
Term (4) ≥ 0 because w ∗ f ∈ W .
B.2 Proof of Lemma
Since strong duality holds, the saddle point ( q π , w ∗ f ) satisfies the KKT conditions.
Then from stationarity, for all ( s, a ) ,
[Equation]
Writing this in matrix form, letting f ′ ( q π ) be shorthand for [ f ′ s,a ( q π ( s, a ))] s,a ∈ R S×A , w ∗ f must satisfy the equality:
[Equation]
B.3 Proof of Proposition
Rearranging the closed form of w ∗ f from Lemma 3 and taking the absolute value of both sides,
[Equation]
Then dividing both sides by d D element-wise, this implies
[Equation]
As the above inequality holds for all ( s, a ) ,
[Equation]
C Proofs for SectionC.1 Proof of Lemma
From the KKT stationarity conditions:
[Equation]
or in matrix form [Equation] letting f ′ ( w π ) be shorthand for [ f ′ s,a ( w π ( s, a ))] s,a ∈ R S×A
[Equation]
Then q ∗ f must satisfy
[Equation]
C.2 Proof of Theorem
The proof is of a similar nature as the proof of Theorem 2 (Appendix B.1).
From Assumption 4 and
Lemma 7, we know that that L wf ( w, q ∗ f ) is an M -strongly convex function in w on the || · || 2 ,η norm.
Since ( w π , q ∗ f ) is the saddle point solution of L wf , from strong convexity we know that the error of (cid:98) w is bounded as
[Equation] [Equation]
where ϵ wstat is given in Lemma 10.
Remark 5.
In Theorem 6 of the main text, there is an additional O ( C wf / √ n ) term in the statistical error ϵ wstat , which would arise if the regularization function E η [ f s,a ( w ( s, a ))] were to be estimated from samples.
However, we state early on in the paper that we assume the regularizer can be calculated exactly, as sampling is a trivial extension.
Correspondingly, the correct expression for the statistical error is:
[Equation]
and, to remain consistent with the rest of the paper, we provide the proof and lemma for this ϵ wstat below.
Lemma 10.
Suppose Assumption 6 holds.
Then for all ( w, q ) ∈ W × Q , w.p. ≥ 1 − δ ,
[Equation]
where ϵ wstat =
(1 + γ ) C w W C w Q [Equation]
+
[Equation]
Proof.
Let l i =
w ( s i , a i )( γq ( s ′ i , π ) − q ( s i , a i )) .
Using Assumption 6,
[Equation]
Then using Hoeffding’s inequality with union bound, w.p. ≥ 1 − δ/
we have that for all w, q ∈
W × Q , (cid:12)(cid:12)(cid:12)(cid:12) (cid:12) n
[Equation]
(cid:88) [Equation] (cid:12)
[Equation]
n
Similarly, for all q ∈ Q , w.p. ≥ 1 − δ/ 2 , (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)nn 0 (cid:88) i = [Equation] − [Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
n Since L wf ( w, q ) =
E η [ f s,a ( w ( s, a ))]
+ E d D [ l i ] + E µ
[ q ( s 0 , π )] , but the first term can be calculated exactly, taking a union bound over the above two inequalities, we have that w.p. ≥ 1 − δ ,
[Equation]
Lemma 11.
Under Assumptions 4, 5, 6, w.p. ≥ 1 − δ ,
[Equation]
Proof of Lemma 11 Letting (cid:98) q ( w ) = arg max q ∈Q (cid:98) L w f ( w, q ) , we decompose the error as follows:
[Equation]
Combining the inequalities gives the result.
We give a brief justification for each term below.
Terms (1) and (4) follow from Lemma 10.
Term (2) ≥ 0 , since q ∗ f ∈ Q .
Term (3) ≥ 0 since w π ∈ W and (cid:98) w = arg max w ∈W (cid:98) L wf ( w, (cid:98) q ( w )) .
Term (5) ≥ 0 since ( w π , q ∗ f ) is a saddle point solution.
D Additional Details of the Experiments D.1 Derivation
We now derive the system of equations for our value function estimation experiments in Section 6.
Letting the regularization function be f s,a ( x ) = 12 x 2 for all ( s, a ) , the objective is
[Equation]
Letting E n denote the empirical average over D for clarity, with empirical samples and the linear classes Q , W , the objective becomes:
[Equation]
Since β ∈ R d , max w ∈W (cid:98) L qf ( q, w )
=
+ ∞ for any q , unless α sets the the second term to 0.
This is satisfied by α such that
[Equation]
However, there may in general be infinite feasible α depending on the linear features and samples.
For our specific linear parameterization of Q , W , the constraints form an underdetermined d × k system of equations, which has infinite solutions.
This is where the regularization term E ν [ α ⊤ ϕ ( s, a ) ϕ
( s, a )
⊤ α ] comes into play.
For any regularizing distribution ν , our method will output a solution that minimizes this term, i.e. that minimizes the norm of q =
Φ ⊤ α on ν .
If ν = 0 , for example, the algorithm will output any feasible point; if ν = 1 / |SA| , the algorithm will output q with smallest L2 norm.
Connection to LSTDQ When using the same linear class for W and Q , the solution to the constraints in Eq.(3) (i.e., ignoring the regularization objective)—if
the solution is unique given matrix invertibility—coincides with LSTDQ [UHJ20].
As mentioned in Section 2, LSTDQ enjoys function-estimation guarantees under matrix invertibility.
In fact, we believe it is possible to extend the analysis even when Q and W use different features of dimensions d and k , respectively; as long as k ≥ d and the matrix in Eq.(3) has full row-rank 9 (i.e., overdetermined ), similar guarantees for
LSTDQ should still hold, though we are not aware of an explicit documentation of this fact.
In contrast, our setup is more challenging as we are in the regime of k < d , and the constraints in Eq.(3) is underdetermined , nullifying the guarantees of LSTDQ.
In such cases, the use of regularization is important for guaranteeing function estimation, as also shown in our experiments.
D.2 Experimental Setup
Feature Design
In total, the tabular environment has 400 state-action values, and we design Φ to aggregate states that correspond to unique entries (within 3 decimal places) of q π .
In Figure 1, (cid:101) Φ is composed of the set of features given by
[Equation]
The first of these two entries is the closed-form solution of w ∗ f given in Lemma 3, and satisfies the realizability requirements of all methods; the second is included for optimization stability.
In Figure 2, we use a model with constant value equal to the average value of q π on the support of p , i.e. q = 1 / |SA| (cid:80) s,a q π ( s, a ) · 1 { p> 0 } .
To maintain realizability when the model is included in the regularization function, (cid:101) Φ is composed of the set
[Equation]
The reason why this preserves realizability is as follows.
When ν is the regularization distribution, and the input model is (cid:101) q
= ( mq π + (1 − m ) q ) ◦ 1 ( p > 0))
for some constant q , the closed-form solution w ∗ f can be expanded as
[Equation]
which implies w ∗ f can be expressed as a linear combination of the three previously defined features.
Solver We solve the linear system using CVXPY with optimizer SCS [DB16; AVDB18].
Environment The Gridwalk is a 10x10 environment with 4 actions corresponding to cardinal directions.
The objective is to reach the goal state (lower right corner).
In each state, the agent receives a reward inversely proportional to its distance from a goal state.
Each trajectory terminates after 100 steps.
The initial states are randomly distributed over the upper half of the grid.
The target policy is defined to be a deterministic optimal policy that always moves towards the goal by first going right, and then down.
To create a strong shift, the behavioral policy is designed to largely explore only the bottom left portion of the grid, providing poor coverage over the target policy and starting states.
Specifically, letting the following probabilities refer to distributions over actions [RIGHT, DOWN, LEFT, UP], the target policy π has distribution [1 , 0 , 0 , 0] over actions until it hits the right wall, then [0 , 1 , 0 , 0] .
The behavior policy takes [0 . 1 , 0 . 4 , 0 . 5 , 0] until it hits the right wall, then takes [0 , 0 . 5 , 0 . 5 , 0] .
E Approximation and Optimization Error
The main results of this paper (Theorems 2, 6) utilize assumptions on realizability (Assumption 2, 5), as well as (implicit) assumptions of perfect optimization.
In this section, we analyze how approximation errors, i.e. when the saddle point solution is not contained in Q × W , and optimization errors affect our error bounds.
Due to the similarity in proofs between value function and weight learning, we provide them only for value function learning; analogous methods can be used to derive similar results for weight learning.
E.1 Finite-sample Guarantees
First, we relax the realizability requirements of Assumption 2.
Define the approximation errors:
[Equation]
ϵ approx : = ϵ approx,q + ϵ approx,w . ϵ approx,q is composed of the worst-case weighted combination of Bellman errors of the best candidate q ∈ Q , as well as the difference between the regularization function at q and q π .
The error ϵ approx,w measures the distance between the best candidate w ∈ W and the saddle point solution w ∗ f by projecting the difference onto the worst-case Bellman error T π q − q .
Remark 6.
To increase intuition of ϵ approx,q , we can relax the difference in regularization terms as E ν [ f s,a ( q ( s, a ))
− f s,a ( q π ( s, a ))] ≤ C qf ′ ||
q π − q || 2 ,ν , which is also the norm upon which the (cid:98) q estimation guarantee is given (Theorem 2) .
Reflecting the nature of the value function estimation task, this states that, even if there is a candidate q ∈ Q with low Bellman error (e.g. if data is sparse), ϵ approx,q will still be large if q is far from q π on the desired distribution ν .
Next, we can also relax the (implicit) assumptions that we obtain the true optima of (5). Let ( (cid:98) q, (cid:98) w ) be the approximate solutions of (5) found by the algorithm.
As before, define (cid:98) w ( q ) := arg max w ∈W (cid:98) L ( w, q ) to be the true empirical maximizer for any q ∈ Q . Note that since we allow for optimization error, it is not necessarily the case that (cid:98) q = arg min q ∈Q (cid:98) L qf ( q, (cid:98) w ( q )) and
(cid:98) w = (cid:98) w (
[Equation]
q ) = arg [Equation] L qf (
[Equation]
q, w ) . Correspondingly, define the following optimization errors: [Equation] ϵ opt,w states that the estimate (cid:98) w should not be too far from the best discriminator in W for (cid:98) q , while ϵ opt,q states that the estimate (cid:98) q should not be too far from the minimax solution.
Using the above definitions, we provide the following generalization of Theorem 2, which accounts for approximation and optimization errors.
Theorem 12.
Under Assumptions 1 and 3, with probability at least 1 − δ ,
[Equation]
where ϵ qstat is given in Theorem 2.
E.2 Proof of Theorem
The proof takes the same overall steps as the proof of Theorem 2 (Appendix B.1), but relies on
Lemma 13 to incorporate the approximation and optimization errors:
[Equation]
Below, we state and prove the helper lemma, which bounds the difference between the Lagrangian objective (4) at the saddle point ( q π , w ∗ f ) and the point ( (cid:98) q, w ∗ f ) :
Lemma 13.
Under Assumptions 1 and 3, w.p. ≥ 1 − δ ,
[Equation]
Proof.
With some abuse of notation (as (cid:101) q, (cid:101) w previously referred to models used with the regularizer), for brevity in this section [Equation] let (cid:101) q be the minimizer of ϵ approx,q and (cid:101) w be the minimizer of ϵ approx,w .
That is,
[Equation]
Decompose the error as follows:
[Equation]
First, (1) holds because ( q π , w ∗ f ) is the saddle point solution of L qf over all q, w ∈ R × R .
The statistical errors in (3) and (6) follow from Lemma 8.
Next, we justify the optimization errors.
For (4),
[Equation]
For (5),
[Equation]
Finally, we justify
[Table]
[Table]
[Equation]
Then since (cid:101) q was chosen to minimize the above expression,
[Equation]
Next we justify (8).
For any w ∈ W and q ∈ Q ,
[Equation]
Then since (cid:101) w was chosen to minimize the RHS of the above inequality,
[Equation]
Combining these inequalities gives the lemma statement.
F Off-Policy Return Estimation
Section 4 demonstrates how q-value estimates (cid:98) q can be obtained [Equation] and Section 5 demonstrates how weight estimates (cid:98) w can be obtained.
The estimates (cid:98) q
and/or
(cid:98) w can additionally be used for downstream off-policy evaluation (OPE) of the policy
’s value J ( π ) , which can be equivalently defined in the following three ways:
[Equation]
With finite samples and estimates (cid:98) q and (cid:98) w approximating q π and w π , respectively, their corresponding off-policy estimators are:
[Equation]
While the OPE estimator (cid:98) J dr ( π ) utilizes both the weights and value functions, (cid:98) J w ( π ) and (cid:98) J q ( π ) utilize only one or the other.
As a result, when (cid:98) q and (cid:98) w are estimated as in Sections 4 and 5, respectively, (cid:98) J w ( π ) and (cid:98) J q ( π ) both inherit their O ( n − 1 / 4 ) sample complexities:
Corollary 14.
Suppose Assumptions 1, 2, and 3 hold, and let
[Equation]
w
[Equation]
q,
[Equation]
Then with probability ≥ 1 − 2 δ ,
[Equation]
where ϵ qeval =
(1 − γ ) C q Q (cid:113) 2log 2 |Q| δ / n 0 , C µ π 0 /ν
=
||
µ π 0 / ν || ∞ , and ϵ qstat is as in Theorem 2.
Corollary 15.
Suppose Assumption 4, 5, and 6 hold, and let
[Equation]
L
[Equation]
( q, w ) .
Then with probability ≥ 1 − 2 δ ,
[Equation]
where ϵ weval =
C w W (cid:113) 2log [Equation] |
[Equation]
n , C d D /η =
∥ d [Equation] / η ∥ ∞ , and ϵ west is as in Theorem 6.
However, when (cid:98) q and (cid:98) w are used together in the doubly robust estimator (cid:98) J dr , their estimation error becomes multiplicative, and (cid:98) J dr ( π ) can achieve the O ( n − 12 ) fast rate of convergence.
In Theorembelow, we present two versions this guarantee.
The first requires no additional assumptions beyond d D > 0 , which we already make (see footnote 5), but involves the largest singular value of I − γP π , which may be difficult to characterize.
The second utilizes an additional assumption, and replaces the singular value with an occupancy ratio, stated below.
The assumption requires that all next states s ′ are also present as states s in transitions of d D (a condition which may reasonably hold in practice), and is also made by [UIJKSX21].
Assumption 7 (Next State Coverage) .
Let d D ( s )
= (cid:80) a d D ( s, a ) be the marginal distribution of states s in d D , and
d Ds ′ ( s ) := (cid:80) s ′ ,a ′ P ( s | s ′ , a ′ )
d D ( s ′ , a ′ ) be the marginal distribution of next states s ′ .
Suppose
[Equation]
Theorem 16.
Suppose Assumption 1, 2, 3, 4, 5, and 6 hold.
Let (cid:98) w and (cid:98) q be estimated from:
[Equation]
Then with probability ≥ 1 − 3 δ ,
[Equation]
If Assumption 7 additionally holds, with probability ≥ 1 − 3 δ ,
[Equation]
where ϵ dreval =
[Equation] / n , σ max denotes the largest singular value, and ϵ qest and ϵ west are as in Theorems 2 and 6.
As the evaluation error ϵ dr eval in Theorem 16 is O ( n − 1 / 2 ) , the sample complexity of doubly robust estimation is rate-limited by ϵ west · ϵ qest , the product of weight and value function estimation errors.
If both functions can be estimated at an O ( n − 1 / 4 ) rate, as is true of our method, then (cid:98) J dr ( π ) attains the overall O ( n − 1 / 2 ) fast rate.
Finally, while Theorem 16 assumes for simplicity that the same Q , W classes are used in both of its optimization problems, it can easily be extended to the case where different pairs of function classes are used as long as the required assumptions hold.
Remark 7 (Comparison to Related Work) .
[YNDLS20] conduct experiments comparing off-policy evaluation using (cid:98) J q ( π ) , (cid:98) J w ( π ) , (cid:98) J dr ( π ) , and generally observe that (cid:98) J dr ( π ) has higher variance and worse performance than either (cid:98) J q ( π ) or (cid:98)
J w ( π ) .
Though at first glance this may appear to contradict
Theorem 16, that is actually not the case; in fact, our theoretical analysis provides insight into why [YNDLS20] may observe such a phenomenon.
In contrast to Theorem 16, when using (cid:98) J dr ( π ) [YNDLS20] utilize saddle point predictions ( (cid:98) q, (cid:98) w ) from either only value function learning or only weight learning, e.g. ( (cid:98) q, (cid:98) w ) =
arg min q ∈Q arg max w ∈W (cid:98)
L qf ( q, w ) that approximates ( q π , w ∗ f ) .
Continuing with this example (and the same applies to weight learning), it is clear from our analysis that (cid:98) w estimated in such a manner may not approximate w π at all, leading to increased estimation error of (cid:98) J dr ( π ) over (cid:98) J q ( π ) .
First, the closed-form solution we have derived for w ∗ f in (Lemma 3) shows that w ∗ f may have a significantly different magnitude from w π .
Second, even if ν and f were chosen such that w ∗ f ≈ w π , as per the reasons stated in Section 4.1, we are not even guaranteed to output (cid:98) w close to w ∗ f since L qf is not regularized in w .
In order to obtain the estimation benefits of doubly robust estimation, our analysis shows that (cid:98) q and (cid:98) w should be separately estimated from their respective optimization problems, then combined in (cid:98) J dr ( π ) .
This is in accordance with similar results from [KU20] and [UIJKSX21].
F.1 Proof of Corollary
Let (cid:101) J
( π ) =
(1 − γ ) E µ
[ (cid:98) q
( s, π )] .
We decompose the error as
[Equation]
First we bound | (cid:98) J ( π ) − (cid:101)
J ( π ) | .
Using Hoeffding’s with union bound, for all q ∈ Q , w.p. ≥ 1 − δ ,
[Equation]
nn
[Equation] qeval [Equation] which implies | (cid:98) J ( π ) − (cid:101) J ( π ) | ≤ ϵ qeval .
For the second term, let C µ π 0 /ν
=
||
µ π 0 /ν || ∞ .
Then w.p. ≥ 1 − δ
[Equation]
using Theorem 2 in the last line.
F.2 Proof of Corollary
Let (cid:101) J
[Equation]
π ) =
E d D [ [Equation] w ( s, a ) r ( s, a )] .
We decompose the error as
[Equation]
For the first term, using Hoeffding’s with union bound, w.p. ≥ 1 − δ , for all w ∈ W ,
[Equation]
which implies | (cid:98) J ( π ) − (cid:101) J ( π ) | ≤ ϵ weval .
For the second term,
[Equation]
w.p. ≥ 1 − δ , using Theorem 6 in the last line.
Taking a union bound over both terms gives the stated result.
F.3 Proof of Theorem
Let (cid:101) J
[Equation] =
[Equation]
+
[Equation]
w ( s, a )( r + (cid:98) q ( s ′ , π ) − (cid:98) q ( s, a ))] .
Again we decompose the error as:
[Equation]
| . For the first term, since E [ (cid:98) J dr ( π )] = (cid:101) J ( π ) , w.p. ≥ 1 − δ we have that ∀ q, w ∈ Q × W ,
[Equation]
For the second term,
[Equation]
where the last equality is due to the fact that (1 − γ ) µ π
= ( I − γP π )( d D · w π ) , and the final inequality is from Cauchy-Schwarz.
We can automatically bound the || (cid:98) w − w π
|| 2 ,d D term using Theorem 6, and it remains to bound ||
( I − γP π )( (cid:98) q − q π )
|| 2 ,d D .
We will consider two cases, first when d D >thus Diag ( d D ) is invertible, and second, when Assumption 7 is satisfied.
In the first case, let D =
Diag ( d D ) , which by assumption is invertible.
Then
[Equation]
in the last line using the fact that the eigenvalues of a matrix A and L − 1 AL are the same for any invertible matrix L .
Thus, denoting the largest singular value of a matrix by σ max ,
[Equation]
Using Theorem 6 and Theorem 2 in the last line to control the errors of (cid:98) w and (cid:98) q in the last line, followed by a union bound over the three inequalities, gives the result.
For the second case, we can directly apply Lemma 17:
[Equation]
and again applying Theorem 6 and Theorem 2 gives the result
[Equation] Lemma 17 uses Assumption 7 to bound the distance in value functions under the transition operator, and is stated and proved below
[Equation]
Lemma 17.
Under Assumption
[Equation]
Proof.
Define ||
P
π
|| 2 ,d D := sup x ̸ =
||
P π x || 2 ,d D / || x
|| 2 ,d D .
Then
[Equation]
It remains to bound ||
P
π
|| 2 ,d D .
For any x ,
[Equation]
This implies that ||
P π || 2 ,d D ≤ (cid:112) C s ′ /s
C π/π D , which gives the stated result.
G Infinite Function Classes
Our results for finite function classes can be easily extended to infinite function classes using covering numbers.
We show that our method value function estimation under infinite function classes achieves the same (cid:101) O ( n − 1 / 4 ) rate as it does under finite function classes (Theorem 2).
The same results also apply to weight function learning using similar proof techniques.
G.1 Finite-sample Guarantees with Infinite Function Classes
First, we define the covering functions used in our results and analysis:
Definition 1 (Covering Number) .
For a function class F , the covering number N ∞ ( ϵ, F ) is defined to be the minimum cardinality of a set F ⊆ F , such that for any f ∈ F , there exists f ∈ F with ∥ f − f ∥ ∞ ≤ ϵ .
Our guarantee for value function learning under infinite function classes is stated below, showing that we achieve the same rate as we do with finite classes.
Theorem 18.
Suppose Assumptions 1, 2, 3 hold.
Then, with probability at least 1 − δ , for ϵ =
B 2 A √ n , || (cid:98) q − q π || 2 ,ν ≤ [Equation] (cid:114) [Equation]
δ n
[Equation]
, where N ∞ ( ϵ, Q ) and N ∞ ( ϵ, W ) are as per Definition 1, and A = 1 + (1 + γ ) C q Q + 2(1 + γ ) C q W and B = C q W (1 + (1 + γ ) C q Q ) .
The proof is given below.
G.2 Proof of Theorem
The statistical error of estimating (cid:98) L ( q, w ) under infinite function classes is the main technical detail of this proof.
Given that, the stated bound on ∥ (cid:98) q − q π ∥ ν can be derived using the same methods (leveraging strong convexity and Lemma 9) as were used in the proofs for value function estimation under finite function classes, i.e. for Theorem 2 (in Appendix B.1) and for Theorem 12 (in
Appendix E.2).
The bound on this statistical error is stated then proved below:
Lemma 19 (Statistical Error under Infinite Function Classes) .
Suppose Assumption 3 holds.
Then, setting ϵ = B 2 A √ n , for any ( q, w ) ∈ Q × W with probability at least 1 − δ ,
[Equation]
n [Equation] where N ∞ ( ϵ, Q ) and N ∞ ( ϵ, W ) are as per Definition 1, and A = 1 + (1 + γ ) C q Q + 2(1 + γ ) C q W and B = C q W (1 + (1 + γ ) C q Q ) .
Proof of Lemma 19.
First, because the regularization term computes E ν [ · ] exactly (not from samples), it has no effect on our bound.
Formally, define the unregularized population Lagrangian to be
[Equation]
and its empirical version to be (cid:98) L 0 ( q, w ) .
Then the LHS of Lemma 19 is equivalent to
[Equation]
so it suffices to bound the statistical error of estimating the unregularized Lagrangian (cid:98) L 0 .
For some (later to-be-specified) ϵ > 0 , let Q be a minimal ϵ -covering of Q in the infinity norm as per Definition 1, that is, |Q| = N ∞ ( ϵ, Q ) .
Let W be defined similarly for W .
Then for any ( q, w ) ∈ Q × W , let ( q, w ) ∈
Q × W be such that ∥ q − q ∥ ∞ ≤ ϵ and ∥ w − w ∥ ∞ ≤ ϵ .
By triangle inequality,
[Equation]
L
[Equation]
Next, define ℓ sas ′ ( q, w ) :
= w ( s, a )( r ( s, a ) + γq ( s ′ , π ) − q ( s, a )) such that L 0 ( q, w ) =
E d D [ ℓ sas ′ ( q, w )] and (cid:98) L 0 ( q, w )
= 1 n (cid:80) ni =1 ℓ s i
a i s ′
i ( q, w ) .
Then we can further upper bound the above as:
[Equation]
Term (T1) can be controlled using the ϵ -covering definition, and (T2) can be controlled using standard concentration methods.
Their respective bounds are provided below, with proofs in the next subsection:
Lemma 20 (Bound for T1) .
Let Q and W be ϵ -coverings of Q and W , respectively, satisfying
Definition 1.
Then for any ( q, w ) ∈ Q × W , there exists (
q, w ) ∈
Q × W such that ∥ q − q ∥ ∞ ≤ ϵ and ∥ w − w ∥ ∞ ≤ ϵ , and if Assumption 3 holds,
[Equation]
with A = 1 + (1 + γ ) C q Q + 2(1 + γ ) C q W .
Lemma 21 (Bound for T2) .
Let Q and W be minimal ϵ -coverings of Q and W , respectively, as in
Definition 1, that is, |Q| = N ∞ ( ϵ, Q ) and |W| =
N ∞ ( ϵ, W ) .
Then if Assumption 3 holds, for any ( q, w ) ∈ Q × W w.p. ≥ 1 − δ ,
[Equation]
n , where B = C q W (1 + (1 + γ ) C q Q ) .
Putting these two bounds together, letting A be as in Lemma 20 and B be as in Lemma 21, we have
[Equation]
Choosing ϵ =
B 2 A √ n gives the final bound:
[Equation]
G.3 Proofs for Helper Lemmas
The proofs of Lemmas 20 and 21 are given below:
Proof of Lemma 20.
For any s, a, s ′ , (since this tuple is fixed, going forward, we drop the s, a, s ′ subscript from ℓ for brevity)
[Equation]
(T3) expresses the error from the covering approximation for w , while (T4) expresses this for q .
First, to bound (T3),
[Equation]
To bound (T4),
[Equation]
Since these two inequalities hold for any sas ′ , combining them directly gives lemma statement.
Proof of Lemma 21.
This is a straightforward application of Hoeffding’s with union bound over
Q , W , akin to the proof of Lemma 8 (which is over Q , W ).