Beyond the Return: Off-policy Function Estimation under User-specified Error-measuring Distributions
Abstract
Off-policy evaluation often refers to two related tasks: estimating the expected return of a policy and estimating its value function (or other functions of interest, such as density ratios).
While recent works on marginalized importance sampling (MIS) show that the former can enjoy provable guarantees under realizable function approximation, the latter is only known to be feasible under much stronger assumptions such as prohibitively expressive discriminators.
In this work, we provide guarantees for off-policy function estimation under only realizability, by imposing proper regularization on the MIS objectives.
Compared to commonly used regularization in MIS, our regularizer is much more flexible and can account for an arbitrary user-specified distribution, under which the learned function will be close to the groundtruth.
We provide exact characterization of the optimal dual solution that needs to be realized by the discriminator class, which determines the data-coverage assumption in the case of value-function learning.
As another surprising observation, the regularizer can be altered to relax the data-coverage requirement, and completely eliminate it in the ideal case with strong side information.
1. Introduction
Off-policy evaluation (OPE) often refers to two related tasks in reinforcement learning (RL): estimating the expected return of a target policy using a dataset collected from a different behavior policy, versus estimating the policy’s value function (or other functions of interest, such as density ratios).
The former is crucial to hyperparameter tuning and verifying the performance of a policy before real-world deployment in offline RL (Voloshin et al., 2019; Paine et al., ) 2020; Zhang & Jiang, 2021).
The latter, on the other hand, plays an important role in (both online and offline) training, often as the subroutine of actor-critic-style algorithms (Lagoudakis & Parr, 2003; Liu et al., 2019), but is also generally more difficult than the former: if an accurate value function is available, one could easily estimate the return by plugging in the initial distribution.
Between the two tasks, the theoretical nature of off-policy return estimation is relatively well understood, especially in terms of the function-approximation assumptions needed for sample-complexity guarantees.
Among the available algorithms, importance sampling (IS) and its variants (Precup et al., 2000; Thomas et al., 2015; Jiang & Li, 2016) do not require any function approximation, but incur exponentialin-horizon variance. Fitted-Q Evaluation (Ernst et al., 2005;
Le et al., 2019) can enjoy polynomial sample complexity under appropriate coverage assumptions, but the guarantee requires the function class to satisfy the strong Bellmancompleteness assumption, i.e. closure under the Bellman operator (Chen & Jiang, 2019; Xie et al., 2021). Marginalized importance sampling (MIS) methods, which have gained significant attention recently (Liu et al., 2018;
Xie et al., 2019; Uehara et al., 2020; Nachum et al., 2019a), use two function classes to simultaneously approximate the value and the density-ratio (or weight) function and optimize minimax objectives.
Notably, it is the only family of methods known to produce accurate return estimates with a polynomial sample complexity, when the function classes only satisfy the relatively weak realizability assumptions (i.e., they contain the true value and weight functions).
In comparison, little is known about off-policy function estimation, and the guarantees are generally less desirable.
Not only do the limitations of IS and FQE on return estimation carry over to this more challenging task, but MIS also loses its major advantage over FQE: despite the somewhat misleading impression left by many prior works, that MIS can handle function estimation the same way as return estimation, 1 MIS for function estimation often requires unrealistic assumptions, such as prohibitively expressive discriminators.
For concreteness, a typical guarantee for function estimation from MIS looks like the following (see e.g., Theorem 4 of Liu et al.
(2018) and Lemmas 1 and 3 of Uehara et al.
Proposition 1 (Function estimation guarantee for MIS, informal) .
Suppose the offline data distribution d D satisfies d D ( s, a ) > 0 , ∀ s, a .
Given value-function class
Q with q π ∈ Q and weight class W = R S×A , q π = arg min q ∈Q max w ∈W L ( w, q ) for some appropriate population loss function L .
To enable the identification of the value function q π , the result requires the discriminator class W to be the space of all possible functions over the state-action space ( W =
R S×A ).
In the finite-sample regime, using such a class incurs a sample complexity that depends on the size of the state-action space, which completely beats the purpose of function approximation.
In addition, these results only hold asymptotically, where the function of interest can be exactly identified in a pointwise manner.
Such an overly strong guarantee is unrealistic in the finite-sample regime, where one can only hope to approximate the function well in an average sense under some distribution , i.e., finite-sample performance guarantees should ideally bound ∥ (cid:98) q − q π ∥ 2 ,ν for the learned (cid:98) q , where ∥·∥ 2 ,ν is ν -weighted 2-norm.
Such fine-grained analyses are non-existent in MIS.
Even in the broader literature, such results not only require Bellman-completeness-type assumptions (Uehara et al., 2021), they also come with some fixed ν (which is not necessarily d D ; see Section 2) and the user has no freedom in choosing ν .
This creates a gap in the literature, as downstream learning algorithms that use offpolicy function estimation as a subroutine often assume the estimation to be accurate under certain specific distributions. For example, in the setting of online policy optimization, Abbasi-Yadkori et al. (2019) require value estimates to be accurate on the occupancy of the (unknown) optimal policy, and (Kakade & Langford, 2002) require them to be accurate on the occupancy of the learning policy at each iteration.
Choosing ν to be the initial state-action distribution is well-suited for off-policy return estimation(see Appendix F). In offline RL, Liu et al. (2019) assume access to weight and value function estimation oracles on ν = d D . To summarize, below are two important open problems on off-policy function estimation:
Is it possible to obtain polynomial 3 sample complexity for off-policy function estimation, using function classes that only satisfy realizability-type assumptions?
Can we specify a distribution ν to the estimation algorithm, such that the learned function will be close to the groundtruth under ν ?
In this work, we answer both open questions in the positive.
By imposing proper regularization on the MIS objectives, we provide off-policy function estimation guarantees under only realizability assumptions on the function classes.
Compared to commonly used regularization in MIS (Nachum et al., 2019a; Nachum & Dai, 2020; Yang et al., 2020), our regularizer is much more flexible and can account for an arbitrary user-specified distribution ν , under which the learned function will be close to the groundtruth.
We provide exact characterization of the optimal dual solution that needs to be realized by the discriminator, which determines the datacoverage assumption in value-function learning.
As another surprising observation, the regularizer can be altered to relax the data-coverage requirement, and in the ideal case completely eliminate it when strong side information is available.
Proof-of-concept experiments are also conducted to validate our theoretical predictions.
2. Related Works
Regularization in MIS The use of regularization is very common in the MIS literature, especially in DICE algorithms (Nachum et al., 2019a;b; Yang et al., 2020).
However, most prior works that consider regularization use tabular derivations and seldom provide finite-sample functionapproximation guarantees on even return estimation, let alone function estimation.
(An exception is the work of
Uehara et al.
(2021), who analyze related estimators under Bellman-completeness-type assumptions; see the next paragraph.)
More importantly, while some existing DICE estimators are subsumed as our special cases when we choose very simple regularizers (see Remark 3 in Section 5), prior works provide very limited understanding in how the choice of regularizers affects learning guarantees, and hence have only considered these na ¨ ıve forms of regularization (
typically state-action-
independent and under d D )—
as different forms of regularization are essentially treated equally under a coarse-grained theory (Yang et al., 2020).
In contrast, we provide much more fine-grained characterization of the effects of regularization, which leads to novel insights about how to design better regularizers.
Fitted-Q Evaluation (FQE) Outside the MIS literature
, one can obtain return and value-function estimation guarantees via FQE (Duan et al., 2020; Chen & Jiang, 2019; Le et al., 2019; Uehara et al., 2021).
However, it is well understood that FQE and related approaches require Bellmancompleteness-type assumptions, such as the function class being closed under the Bellman operator.
Even putting aside the difference between completeness vs. realizability, we al- low for a user-specified error-measuring distribution, which is not available in FQE or any other existing method.
The only distribution these methods are aware of is the data distribution d D , and even so, FQE and variants rarely provide guarantees on ∥ (cid:98) q − q π ∥ 2 ,d D , but often on the Bellman error (e.g., ∥ (cid:98) q − T π (cid:98) q ∥ 2 ,d D ) instead (Uehara et al., 2021), and obtaining guarantees on a distribution of interest often requires multiple indirect translations and loose relaxations.
LSTDQ
Our analyses focus on general function approximation.
When restricted to linear classes, function estimation guarantees for q π under d D can be obtained by LSTDQ methods (Lagoudakis & Parr, 2003; Bertsekas & Yu, 2009;
Dann et al., 2014) when the function class only satisfies realizability of q π (Perdomo et al., 2022).
However, this requires an additional matrix invertibility condition (see Assumption 3 of Perdomo et al.
, and it is still unclear what this condition corresponds to in general function approximation 4 .
Moreover, many general methods—including MIS (Uehara et al., 2020) and other minimax methods (Antos et al., 2008; Xie et al., 2021)—coincide with LSTDQ in the linear case, so the aforementioned results can be viewed as a specialized analysis leveraging the properties of linear classes.
PRO-RL (Zhan et al., 2022)
Our key proof techniques are adapted from Zhan et al.
(2022), whose goal is offline policy learning.
They learn the importance weight function w π for a near-optimal π , and provide ∥ (cid:98) w − w π ∥ 2 ,d D guarantees as an intermediate result.
Despite using similar technical tools, our most interesting and surprising results are in the value-function estimation setting, which is not considered by Zhan et al.
Our novel algorithmic insights, such as incorporating error-measuring distributions and approximate models in the regularizers, are also potentially useful in Zhan et al.
(2022)’s policy learning setting.
Our analyses also reveal a number of important differences between
OPE and offline policy learning, which will be discussed in
Appendix A.
3. Preliminaries
We consider off-policy evaluation (OPE) in Markov Decision Processes (MDPs).
An MDP is specified by its state space S , action space A , transition dynamics P : S ×
A → ∆( S ) ( ∆( · ) is the probability simplex), reward function
R : S ×
A → ∆([0 , 1]) , discount factor γ ∈ [0 , 1) , and an initial state distribution µ 0 ∈ ∆( S ) .
We assume S and A are finite and discrete, but their cardinalities can be arbitrarily large.
Given a target policy π : S → ∆( A ) , a random trajectory s 0 , a 0 , r 0 , s 1 , a 1 , r 1 , . . .
can be generated as s 0 ∼ µ 0 , a t ∼ π ( ·| s t ) , r t ∼ R ( ·| s t , a t ) , s t +1 ∼ P ( ·| s t , a t ) , ∀ t ≥ 0 ; we use E π and P π to refer to expectation and probability under such a distribution.
The expected discounted return (or simply return) of π is J ( π ) :=
E π [ (cid:80) t γ t r t ] .
The
Q-value function of π is the unique solution of the Bellman equations q π = T
π q π , with the Bellman operator T π :
R S×A → R S×A defined as ∀ q ∈ R S×A , ( T π q )( s, a ) :=
E r ∼ R ( ·| s,a )
[ r ] + γ ( P π q )( s, a ) .
Here P π ∈ R |S×A|×|S×A| is the state-action transition operator of π , defined as ( P π q )( s, a ) :
=
E s ′ ∼ P ( ·| s,a ) ,a ′ ∼ π ( ·| s ′ )
[ q ( s ′ , a ′ )] .
Functions over S × A (such as q ) are also treated as |S × A| - dimensional vectors interchangeably.
In OPE, we want to estimate q π and other functions of interest based on a historical dataset collected by a possibly different policy.
As a standard simplification, we assume that the offline dataset consisting of n i.i.d.
tuples { ( s i , a i , r i , s ′ i ) } ni =1 sampled as ( s i , a i ) ∼ d D , r =
R ( s i , a i ) , and s ′ i ∼ P ( ·| s i , a i ) .
We call d D the (offline) data distribution.
As another function of interest, the (marginalized importance) weight function w π is defined as w π ( s, a ) :
=
d π ( s, a ) /d D ( s, a ) , where d π ( s, a ) =
(1 − γ ) (cid:80) ∞
t =
γ t P π [ s t = s, a t = a ] is the discounted stateaction occupancy of π .
For technical convenience we assume d D ( s, a ) > 0 ∀ s, a , so that quantities like w π are always well defined and finite.
5 Similarly to q π , w π also satisfies a recursive equation, inherited from the Bellman flow equation for d π : d π = (1 − γ )
µ π 0 + γ (cid:101) P π d π , where ( s, a ) ∼ µ π 0 ⇔ s ∼ µ 0 , a ∼ π ( ·| s ) is the initial state-action distribution, and (cid:101) P π
=
( P π ) ⊤ is the transpose of the transition matrix.
Function Approximation We will use function classes
Q and W to approximate q π and w π , respectively.
We assume finite Q and W , and extension to infinite classes under appropriate complexity measures (e.g., covering number) is routine and orthogonal to the main insights of the paper.
Additional Notation ∥ · ∥ 2 ,ν := (cid:112) E ν [( · ) 2 ] is the weighted 2-norm of a function under distribution ν .
We also use a standard shorthand f
( s, π ) :
= E a ∼ π ( ·| s )
[ f ( s, a )] .
u ◦ v between two vectors u and v of the same dimension is elementwise multiplication, and u/v is elementwise division.
4. Value-function Estimation
In this section we show how to estimate (cid:98) q ≈ q π with guarantees on ∥ (cid:98) q − q π ∥ 2 ,ν for a user-specified ν , and identify the assumptions under which provable sample-complexity guarantees can be obtained.
We begin with the familiar
Bellman equations, that q π is the unique solution to:
[Equation]
While the above set of equations uniquely determines q = q π , this is only true if we can enforce all the |S × A| constraints, which is intractable in large state-space problems.
In fact, even estimating (a candidate q ’s violation of) a single constraint is infeasible as that requires sampling from the same state multiple times, which is related to the infamous double-sampling problem (Baird, 1995).
To overcome this challenge, prior MIS works often relax (1) by taking a weighted combination of these equations, e.g.
[Equation]
In words, instead of enforcing |S × A| equations, we only enforce their linear combinations; the linear coefficients are d D ( s, a ) · w ( s, a ) , and w belongs to a class W with limited statistical capacity to enable sample-efficient estimation.
While each constraint in (2) now be efficiently checked on data, this comes with a big cost that a solution to (2) is not necessarily q π .
Prior works handle this dilemma by aiming lower: instead of learning (cid:98) q ≈ q π , the problem becomes tractable if we only aim to learn (cid:98) q that can approximate the policy’s return, i.e. E s ∼ µ
[ (cid:98) q
( s, π )]
≈ J ( π ) =
E s ∼ µ
[ q π ( s, π )] .
While Uehara et al.
(2020) show that this is possible under w π ∈ W , they also show explicit counterexamples where (cid:98) q ̸ = q π even with infinite data.
As a result, how to estimate (cid:98) q ≈ q π under comparable assumptions (instead of the prohibitive W = R S×A as in Proposition 1) is still an open problem.
4.1. Estimator
We now describe our approach to this problem.
Recall that the goal is to obtain error bounds for ∥ (cid:98) q − q π ∥ 2 ,ν for some distribution ν ∈ ∆( S × A ) provided by the user.
Note that we do not require information about r and s ′ that are generated after ( s, a ) ∼ ν
and only care about the ( s, a ) marginal itself, so the user can pick ν without knowing the transition and the reward functions of the MDP.
We assume that ν is given in a way that we can take its expectation E ( s,a ) ∼ ν
[( · )] ; the extension to the case where ν is given via samples is straightforward.
To achieve this goal, we first turn (1) into an equivalent constrained convex program : given a collection of strongly convex and differentiable functions f = { f s,a : R → R } s,a — we will later discuss its choice—consider
[Equation]
The constraints here are the same as (1).
Since (1) uniquely determines q = q π , the feasible space of (3) is a singleton , so we can impose any objective function on top of these constraints, and it will not change the optimal solution (which is always q π , the only feasible point). Here, we use an objective function of the form E ( s,a ) ∼ ν [ f s,a ( q ( s, a ))] , where ν is the user-provided distribution of interest. Choosing f s,a ( q ( s, a ))
= 0 ∀ s, a , for example, recovers MQL in Uehara et al. With this choice, however, as shown in Proposition 1, q π cannot be identified without prohibitive assumptions. As we will see, other choices of f will serve as important regularizers in the function-approximation setting, and will be crucial for our function estimation guarantees.
Remark 1 ( ( s, a ) -dependence of f ) .
Regularizers in prior works are ( s, a ) - independent (Nachum et al., 2019a;
Yang et al., 2020; Zhan et al., 2022).
As we will see in Section 4.3, allowing for ( s, a ) -dependence is very important for designing regularizers with improved guarantees and performances.
We now rewrite (3) in its Lagrangian form, with d D ◦ w serving the role of dual variables:
[Equation]
Finally, our actual estimator approximates (4) via finitesample approximation of the population loss L qf , and searches over restricted function classes Q and W for q and w , respectively:
[Equation]
where (cid:98) L qf ( q, w ) :=
E ν [ f s,a ( q ( s, a ))] + 1 n (cid:80) ni [Equation] 1 w ( s i , a i ) ( r i + γq ( s
[Equation]
i , π ) − q ( s i , a i )) . Intuition for identification Before giving the detailed finite-sample analysis, we provide some high-level intuitions for why we can obtain the desired guarantee on ∥ (cid:98) q − q π ∥ 2 ,ν .
Note that (4) is structurally similar to (2), and we still cannot verify the Bellman equation for q π in a per-state-action manner, so the caveat of (2) seems to remain; why can we identify q π under ν ?
The key here is to show that it suffices to check the loss function L qf only under a special choice of w (as opposed to all of R S×A ).
Importantly, this special w is not w = w π ; rather, it is the function w ∗ f in the saddle-point solution of our regularized objective, ( q π , w ∗ f ) =
arg min q arg max w L qf ( q, w ) , for which we later provide the closed form.
As long as w ∗ f ∈ W —even if W is extremely “simple” and contains nothing but w ∗ f —we can identify q π .
Intuitively, w π should not appear in our analysis at all: w π is defined with respect to the initial distribution of the MDP µ 0 , which has nothing to do with our goal of bounding ∥ (cid:98) q
− q π ∥ 2 ,ν .
To see that, it is instructive to consider the special case of
W = { w ∗ f } and the limit of infinite data.
In this case, our estimator becomes arg min q ∈Q L qf ( q, w ∗ f ) .
By the definition of saddle point:
[Equation]
While this shows that q π is a minimizer of the loss, it does not imply that it is a unique minimizer.
However, identification immediately follows from the convexity brought by regularization: since f : R → R is strongly convex, q → E ν [ f s,a ( q ( s, a ))] as a mapping from R S×A to R is strongly convex under ∥ · ∥ 2 ,ν (see Lemma 7 in Appendix B for a formal statement and proof), and L qf ( q, w ∗ f ) inherits such convexity since the other terms are affine in q .
It is then obvious that q π is the unique minimizer of L qf ( q π , w ∗ f ) up to ∥ · ∥ 2 ,ν , that is, any minimizer of L qf must agree with q π on ( s, a ) pairs supported on ν .
Our finite-sample analysis below shows that the above reasoning is robust to finite-sample errors and having functions other than w ∗ f in W .
4.2. Finite-sample Guarantees
In this subsection we state the formal guarantee of our estimator for q π , and the assumptions under which it holds.
We start with the condition on the regularization function f :
Assumption 1 (Strong convexity of f ) .
Assume f s,a : R → R is nonnegative, differentiable, and M q -strongly convex for each s ∈ S , a ∈ A .
In addition, assume both f s,a and f ′ s,a take finite values for any finite input.
This assumption can be satisfied by a simple choice of f s,a ( x ) = 12 x 2 , which is independent of ( s, a ) and yields
M q = 1 .
Alternative choices of f are discussed in Section 4.3.
Next are the realizability and boundedness of W and Q :
Assumption 2 (Realizability) .
Suppose w ∗ f ∈ W , q π ∈ Q .
Assumption 3 (Boundedness of W and Q ) .
Suppose W and Q are bounded, that is,
[Equation]
As a remark, Assumption 2 implicitly assumes the existence of w ∗ f .
As we will see in Section 4.3, the existence and finiteness of w ∗ f is automatically guaranteed given the finiteness of f ′ s,a (Assumption 1) and d D ( s, a ) > 0 ∀ s, a .
More importantly, Assumptions 2 and 3 together imply that ∥ q π ∥ ∞ ≤ C q Q and ∥ w ∗ f ∥ ∞ ≤ C q W , which puts constraints on how small C q Q and C q W can be.
For example, it is common to assume that C q Q = 11 − γ , i.e., the maximum possible return when rewards are bounded in [0 , 1] , and this way ∥ q π ∥ ∞ ≤ C q Q will hold automatically.
The magnitude of ∥ w ∗ f ∥ ∞ and C q W , however, is more nuanced and interesting, and we defer the discussion to Section 4.3.
Now we are ready to state the main guarantee for identifying [Equation]
All proofs of this section can be found in Appendix B.
Theorem 2.
Suppose Assumptions 1, 2, 3 hold.
Then, with probability at least 1 − δ ,
[Equation]
where ϵ qstat =
(cid:0) C q W + (1 + γ ) C q W C q Q (cid:1) (cid:113) 2log 2 |W||Q| δ / n .
Theorem 2 shows the desired bound on ∥ (cid:98) q − q π ∥ 2 ,ν , which depends on the magnitude of functions in W and Q as well as their logarithmic cardinalities, which are standard measures of statistical complexity for finite classes.
One notable weakness is the O ( n − 1 / 4 ) slow rate; this is due to translating the ϵ qstat =
O ( n − 1 / 2 ) deviation between L and (cid:98) L into ∥ (cid:98) q − q π ∥ 2 ,ν via a convexity argument, which takes square root of the error.
The possibility of and obstacles to obtaining an O ( n − 1 / 2 ) rate will be discussed in Section 7.
4.3. On the Closed Form of w ∗ f and the Data Coverage Assumptions
One unusual aspect of our guarantees in Section 4.2 is that we do not make any explicit data coverage assumptions, yet such assumptions are known to be necessary even for return estimation (typically the boundedness of w π =
d π /d D ).
Indeed, our data-coverage assumption is implicit in Assumptions 2 and 3, which require ∥ w ∗ f ∥ ∞ ≤ C q W < ∞ .
If data fails to provide sufficient coverage, ∥ w ∗ f ∥ ∞ will be large and our bound in Theorem 2 will suffer due to a large value of C q W .
To make the data coverage assumption explicit, we provide the closed-form expression of w ∗ f :
Lemma 3.
The saddle point of (4) is ( q π , w ∗ f ) =
arg min q arg max w L qf ( q, w ) , where
[Equation]
Here f ′ ( q π ) is the shorthand for [ f ′ s,a ( q π ( s, a ))] s,a ∈
R S×A .
The closed-form expression in (6) looks very much like a density ratio: if we replace ν ◦ f ′ ( q π ) with µ π 0 , we have ( I − γ (cid:101) P π ) − 1 µ π 0 =
d π / (1 − γ ) , and the expression would be the ratio between d π and d D (up to a horizon factor).
Therefore, w ∗ f can be viewed as the density ratio of π against d D when π starts from the “fake” initial distribution ν ◦ f ′ ( q π ) .
However, ν ◦ f ′ ( q π ) is in general not a valid distribution, as it is not necessarily normalized or even non-negative, making ∥ w ∗ f ∥ ∞ difficult to intuit.
Below we give relaxations of ∥ w ∗ f ∥ ∞ , which are more interpretable and give novel insights into how to relax the data-coverage assumption via tweaking f .
Proposition
[Equation]
[Equation]
where d πν is the discounted state-action occupancy of π under ν as the initial state-action distribution.
The proposition states that ∥ w ∗ f ∥ ∞ can be bounded if data provides sufficient coverage over d πν , and if f ′ ( q π ) is bounded.
The former shows that d D needs to cover not only ν , but also state-action pairs reachable by π starting from ν .
The latter is easily satisfied, and can be bounded again for concrete choices of f , e.g. ∥ f ′ ( q π ) ∥ ∞
≤ ∥ q π ∥ ∞ ≤ 11 − γ for f s,a ( x ) = 12 x 2 .
Designing f to relax the coverage assumption Lemmashows that the coverage assumption ( ∥ w ∗ f ∥ ∞ ) depends on f (or rather its derivative f ′ ), which opens up the possibility of properly designing f to relax it.
In fact, we could completely eliminate the coverage assumption if we could set f ′ ( q π ) = 0 , but that would require unrealistically strong side information.
As a concrete example, consider f s,a ( x ) = 12 ( x − q π
( s, a )) 2 , and it is easy to verify that f ′ s,a ( q π ( s, a )) =
x − q π ( s, a ) | x = q π ( s,a ) = 0 .
Compared to f s,a ( x ) = 12 x 2 , the new f essentially adds a 1st-order term q π ( s, a ) · x to change w ∗ f , while leaving the convexity required by Assumption 1 intact, which only depends on the 2nd-order term 12 x 2 .
Of course, this is not a viable choice of f in practice as it requires knowledge of q π , which is precisely our learning target.
While the reason f s,a ( x ) = 12 ( x − q π ( s, a )) 2 can eliminate the coverage requirement is obvious retrospectively ( q π already minimizes E ν [ f ( q )] even without any data), our analyses apply much more generally and characterize the effects of arbitrary f on the coverage assumption.
Inspired by this example, we can consider practically feasible choices such as f s,a ( x ) = 12 ( x − (cid:101) q
( s, a ))
2 , where (cid:101) q is an approximation of q π obtained by other means, e.g. a guess based on domain knowledge.
If (cid:101) q ≈ q π , our estimator enjoys significantly relaxed coverage requirements.
But even if (cid:101) q is a poor approximation of q π , it does not affect our estimation guarantees as long as the condition implied by Propositionis satisfied.
(In fact, f s,a ( x ) = 12 x 2 is a special case of (cid:101) q ≡ 0 .)
Such a use of approximate models is similar to how doubly robust estimators (Dud´ık et al., 2011; Jiang & Li, 2016; Thomas & Brunskill, 2016) enjoy reduced variance given an accurate model, and remain unbiased even if the approximate model is arbitrarily poor.
We will also empirically evaluate the effectiveness of this idea in Section 6.
5. Weight-function Estimation
Similar to value-function estimation, our methodology can also be applied to estimate the weight function w π .
Due to the similarity with Section 4 in the high-level spirit, we will be concise in this section and only explain in detail when there is a conceptual difference from Section 4.
Some notations (such as the function classes W and Q ) will be abused, but we emphasize that this section considers a different learning task than Section 4, so they should be viewed as different objects (e.g., the realizability assumptions for
W and Q below will be different from those in Section 4).
As before, we assume that the user provides a distributionη ∈ ∆( S × A ) and our goal is to develop an estimator with guarantees on ∥ (cid:98) w − w π ∥ 2 ,η .
Analogous to Section 4, consider
[Equation]
Here f = { f s,a } s,a will need to satisfy similar assumptions as in Section 4.
The constraints are the Bellman flow equations with a change of variable d ( s, a ) =
d D ( s, a ) · w ( s, a ) .
Their unique solution is d ( s, a ) =
d π ( s, a ) (and hence w ( s, a ) =
d π ( s, a ) /d D
( s, a ) ), thus the feasible space is again a singleton, and the objective does not alter the optimal solution.
We then use dual variables q to rewrite (7) in its Lagrangian form:
[Equation]
, a [Equation] We approximate the saddle-point solutions by optimizing the empirical loss (cid:98) L wf over restricted function classes W , Q :
[Equation]
where (cid:98) L wf ( q, w ) :=
E
[Equation]
[ f s,
[Equation]
w [Equation] [Equation] and { s j } n 0 j =1 is a separate dataset sampled i.i.d.
from µ 0 to provide information about initial distribution.
We provide the closed-form expression for the saddle point of L wf below, which resembles the Q-function for a proxy reward function f ′ ( w π ) ◦ η/d D .
Lemma 5.
The closed form solutions of (8) are ( w π , q ∗ f ) =
arg min w arg max q L wf ( q, w ) , where
[Equation]
Remark 2 (Data Coverage Assumption) .
As we will see, the only data coverage assumption we need is the boundedness of w π = d
π /d D .
Since w π is the function of interest and practical algorithms can only output functions of wellbounded ranges, such an assumption is an essential part of the learning task itself and hardly an additional requirement.
Moreover, unlike Section 4, changing f here will not affect the data-coverage assumption, though it still alters q ∗ f , and a properly chosen f (e.g., with f ′ ( w π ) ≈ 0 ) can still result in a q ∗ f with small magnitude and thus make learning easier.
Remark 3 (Connection to DualDICE) .
We can recover
DualDICE (Nachum et al., 2019a) by choosing f s,a ( x ) = 12 x 2 and ν = d D .
Despite producing the same estimator, the derivations and assumptions under which the two works analyze the estimator are different.
Their Theorem 2 only provides return estimation guarantees, and depends on an implicit assumption of highly expressive function classessimilar to Proposition 1.
Moreover, they do not characterize how the choice of f can affect the learning guarantees (their f is ( s, a ) -independent).
This is one of the main insights of our paper and leads to the discovery of more practical regularizers, e.g. f s,a ( x ) = 12 ( x − ˜ w ( s, a )) 2 with model (cid:101) w .
Below we present the assumptions, then learning guarantee for (cid:98) w .
Assumption 4 (Strongly Convex Objective) .
Suppose for all s, a , f s,a is differentiable, non-negative, and M wstrongly convex.
Further, suppose f s,a and its derivative take finite values on any finite inputs, and let C wf := max w ∈W || f ( w ) || ∞ .
Assumption 5 (Realizability) .
Suppose w π ∈ W , q ∗ f ∈ Q .
Assumption 6 (Bounded W and Q ) .
Let C w W := max w ∈W || w || ∞ and C w Q :=
max q ∈Q || q || ∞ .
Suppose W and Q are bounded function classes, that is, C w W < ∞ and
C w Q < ∞ .
Theorem 6.
Suppose Assumptions 4, 5, 6, hold.
Then w.p.
≥ [Equation] − [Equation]
[Equation] where ϵ wstat =
(1 + γ ) C w W C w Q (cid:113) 2log [Equation] Q
[Equation]
6. Experiments
We now provide experimental results to verify our theoretical predictions and insights.
As Yang et al.
(2020) have performed extensive experiments on return estimation with simple regularization ( f s,a ( x ) = 12 x 2 ), we focus on the task of q π estimation, and the following two questions unique to our work:
Q1.
When the goal is to minimize ∥ (cid:98) q − q π ∥ 2 ,ν , how much benefit does regularizing with ν bring in practice, compared to regularizing with other distributions (or no regularization at all)?
Q2.
Can incorporating (even relatively poor) models in regularization (e.g., f s,a ( x ) = 12 ( x − (cid:101) q ( s, a )) 2 from Section 4.3) improve estimation?
Setup We study these questions in a large tabular Gridwalk environment (Nachum et al., 2019a; Yang et al., 2020), with a deterministic target policy π that is optimal, and a behavior policy that provides limited coverage over the target policy; see Appendix D for details.
To mimic the identification challenges associated with restricted function classes, we use a linear function class Q = { Φ ⊤ α : α ∈ R d } and discriminator class W = { (cid:101)
Φ ⊤ β : β ∈ R k } , where k < d ≪ |S × A| .
The features Φ ∈ R |S×A|× d , (cid:101)
Φ ∈ R
|S×A|× k are chosen to satisfy the realizability assumptions of all estimators.
Under linear classes, our estimator ((5)) becomes a convex optimization problem with d variables and k linear constraints, and can be solved by standard packages.
This allows us to avoid difficult minimax optimization—which is still an open problem in the MIS literature—and focus on the statistical behaviors of our estimators, which is what our theoretical predictions are about.
Remark 4.
When no regularization is used, our estimator coincides with MQL (Uehara et al., 2020).
If we further had (cid:101) Φ =
Φ , the estimator would coincide with LSTDQ.
While Section 2 mentioned that LSTDQ enjoys functionestimation guarantees (Perdomo et al., 2022) (and folklore suggests they extend to (cid:101) Φ ̸ = Φ ), the guarantee only holds in the regime of k ≥ d , i.e., the k linear constraints are over-determined .
In our case, however, we have underdetermined constraints ( k < d ), creating a more challenging learning task (which our theory can handle) where LSTDQ’s guarantees do not apply.
Choice of Distributions We consider a set of diverse distri- [Figure] [Figure] [Figure]
log | J ( ) J ( )| -0.
log q q 2, d D -0.
log q q 2, U -0.
log q q 2, p none d D
U d p
[Figure]
Uniform Model p Model
[Figure]
p
[Figure]
butions V = { d D , µ π 0 , d π , U, p } , where U is uniform over
S × A and p ∝ ( d π ◦ I [ w π > 50]) .
The distribution p isolates the least-covered states reached by π , which makes learning an accurate Q-function on ν a harder task.
Results for Q
We use a default regularizer f = 12 xwith different regularizing distributions ν ∈ V , and measure ∥ (cid:98) q − q π ∥ 2 ,ν ′ for different ν ′ .
The results are shown in
Figure 1, and exhibit the expected trend.
For example, regularizing with ν = p performs poorly when the error is measured under ν ′ =
d D and U due to the large mismatch between ν and ν ′ .
However, when ν ′ =
p (rightmost panel), regularizing with ν = p significantly outperforms others.
Similar behaviors can also be observed on U , though they are certainly not absolute
(e.g., ν =
d D does not do very well on ν ′ =
d D ), which suggests potential directions for more refined theory.
Moreover, using no regularization (“none”) generally does not perform well for any ν ′ , but still manages to achieve a high accuracy for return estimation
J ( π ) , which is consistent with prior theory (Uehara et al., 2020) that return estimation does not require regularization.
Results for Q
We now use f s,a ( x ) = 12 ( x − (cid:101) q ( s, a )) 2 with different (cid:101) q to verify how the quality of (cid:101) q affects estimation accuracy.
We first consider a “uniform model” (cid:101) q = mq π
+ (1 − m ) q , where q is a constant and m ∈ [0 , 1] controls the quality (cid:101) q .
As shown in Panels 1 & 3 of Figure 2, our estimator’s accuracy generally improves with a better (cid:101)
q
(i.e., as m increases).
Moreover, equipping (cid:101) q with an appropriate regularizing distribution ν (e.g., ν = U for both panels) can significantly outperform no regularization, even with a very poor (cid:101) q
(e.g., m = 0 . 1 ).
It also outperforms the model prediction itself (i.e., (cid:98) q
= (cid:101) q ), showing that the improvement is not from simply taking predictions from (cid:98) q , but instead from using the regularization to better identify q π from data.
The previous model’s quality is uniform across S × A .
We then consider a scenario where (cid:101) q is zeroed out outside p ’s support, making it only a good approximation of q π on p .
In this case, we see that regularization cannot benefit much from the model when the error is measured on ν ′ =
U (Panel 2), but when ν ′ =
ν = p (Panel 4), regularization can still bring benefits, as expected from our theory.
7. Discussion and Conclusion
In this paper, we showed that proper regularization can yield function-estimation guarantees for MIS methods under only realizable function approximation.
Our results identify when and why accurate off-policy estimates of weight and value functions, which play important roles in larger reinforcement learning algorithms, can be obtained, and provide insight into how different regularizers lead to better estimates.
Compared to prior works, our regularizer is more flexible and can accommodate a user-specified errormeasuring distribution.
Further theoretical investigation provides fine-grained characterization of how the choice of regularization affects learning guarantees, which leads to the discovery of regularizers that incorporate approximate models (such as (cid:101) q ).
While the superiority of such regularizers is perhaps obvious retrospectively, it is not allowed in the prior works’ derivation that assumes ( s, a ) -independent regularization, and our theoretical results provide a deep understanding for even more general regularization schemes.
In Appendix A, we provide further discussions on two topics: (1) the barriers to obtaining a faster O ( n − 1 / 2 ) rate, and (2) comparison to Zhan et al.
(2022) reveals interesting differences between off-policy function estimation and policy learning, and insights in this paper may also be useful for the policy learning task.
In Appendix E we discuss robustness to approximation and optimization errors.
Finally, in
Appendix F, we discuss how estimated functions can be used in downstream off-policy evaluation tasks, and provide corresponding estimation guarantees.
For the aforementioned off-policy evaluation and off-policy learning (Liu et al., 2019) tasks, function estimation is (naturally) generally required to be accurate on ν = d D or µ π and η = d D (Liu et al., 2018; Perdomo et al., 2022; Liu et al., 2019), and samples or exact distributions are accessible to the user.
To this end, our method provides a concise answer to a missing piece of off-policy algorithms.
However, the interaction of our method with online training methods is less clear. As mentioned in the introduction, online algorithms using off-policy function estimation as a subroutine, such as (Kakade & Langford, 2002; AbbasiYadkori et al., 2019), may require the estimates to be accurate on unknown distributions such as d π or d π ∗ (where π ∗ is the optimal policy), which may not be immediately accessible to the user. If the algorithm or user desires a guarantee on ν = d π , with access to a simulator, they may be able to roll out samples of d π to use with our method (which is compatible with sampling). If samples from a replay buffer are used to estimate functions of π , our method would provide estimation guarantees over ν corresponding to the distribution of the buffer.
Under appropriate coverage assumptions, a change of distribution could then be used to convert these guarantees over ν to a guarantee over d π .
In other cases, the user may be able to use domain knowledge to “guess” a distribution ν close to or covering the unknown distribution of interest.
Then our guarantees for function estimation over ν could similarly, with a change of distribution, be converted to guarantees on the true distribution of interest.
To this end, an important avenue of future work involves a thorough investigation of how our off-policy function estimation method interacts with such online learning methods, their assumptions, and their guarantees.
Acknowledgment
The authors thank Wenhao Zhan and Jason Lee for valuable discussions during the early phase of the project.
NJ acknowledges funding support from ARL Cooperative Agreement W911NF-17-2-0196, NSF IIS-2112471, NSF CAREER IIS-2141781, and Adobe Data Science Research
Award. A. Further Discussions Faster rate One weakness of our result is the O ( n − 1 / 4 ) slow rate of estimation.
While O ( n − 1 / 2 ) generalization error bounds for related stochastic saddle point exist (Zhang et al., 2021), they only apply to strongly-convex-strongly-concave problems, whereas our problem is strongly-convex-non-strongly-concave ( L qf is affine in w and L wf is affine in q ), making the result not directly applicable.
One immediate idea is to introduce dual regularization to make our objectives also strongly concave in the discriminator.
However, while primal regularization does not change the feasible space and guarantees that the learned function will be q π (or w π , respectively), dual regularization does change the optimal solution, introducing a bias.
This leads to a trade-off between the improvement in error bounds due to strong concavity and the additional bias, and our preliminary investigation shows that an optimal trade-off between the two sources of errors still leads to an O ( n − 1 / 4 ) rate.
Therefore, improving the rate (if it is possible at all) will require novel technical tools for the generalization analyses of strongly-convex-non-strongly-concave stochastic saddle point problems, which will be an interesting future direction.
On a related note, while the rate for estimating q π and w π is only O ( n − 1 / 4 ) , we can combine them in a doubly robust form to get O ( n − 1 / 2 ) rate for return estimation by careful choices of the regularizing distributions ν and η ; see Appendix F for details.
Comparison to off-policy learning As mentioned earlier, our results are enabled by technical tools adapted from Zhan et al.
(2022), whose work focuses on off-policy policy learning and learns w π for a near-optimal π that is accurate under d D as an intermediate step.
While most of our surprising observations are in the value-function learning scenario (Section 4), comparing our guarantee for learning w π (Section 5) to that of Zhan et al.
(2022) still yields interesting observations about the difference between off-policy evaluation and learning.
Most notably, we do not need to control the strength of regularization in (3), since the feasible space is a singleton and there is no objective before we introduce E ν [ f ( q )] .
In contrast, the feasible space is not a singleton in Zhan et al.
(it is the space of all possible occupancies) and there is already a return optimization objective, so Zhan et al.
(2022) need to carefully control the strength of their regularization.
As a consequence, Zhan et al.
(2022) obtain O ( n − 1 / 6 ) rate, showing how off-policy learning is potentially more difficult than off-policy function estimation.
Another interesting difference is related to our exact characterization of w ∗ f and q ∗ f : Zhan et al.
(2022) do not have a closed-form expression for their optimal dual solution.
Such a lack of direct characterization leads to requiring additional assumptions to guarantee the boundedness of such variables (see their Assumptions 11 and 12), which is not a problem in our setting.
Finally, our analyses lead to novel algorithmic ideas such as using state-action-dependent regularizers and incorporating approximate models in the regularizers, which are potentially also useful for policy learning.
B. Proofs for SectionB.1. Proof of Theorem
From Assumption 1 and Lemma 7, we know that the regularization function E ν [ f s,a ( q ( s, a ))] is an M -strongly convex function in q on the ∥ · ∥ 2 ,ν norm.
Now consider L qf ( q, w ∗ f ) , the Lagrangian function (4) at the optimal discriminator w ∗ f .
Since L qf ( q, w ∗ f ) is composed of the regularization function plus terms that are linear in q , L qf ( q, w ∗ f ) is also an M -strongly convex function in q .
As ( q π , w ∗ f ) is the saddle point solution of L qf , we know q π = arg min q L qf ( q, w ∗ f ) .
Then from the strong convexity of L qf , || (cid:98) q − q π || 2 ,ν ≤ (cid:118)(cid:117)(cid:117)(cid:116) 2 (cid:16) L qf (
[Equation]
where ϵ qstat is given in Lemma 8.
We provide the helper lemmas and their proofs below:
Lemma 7.
Suppose f s,a : R → R is M -strongly convex.
Then E ν [ f s,a ( q ( s, a ))] : R | SA | → R is M -strongly convex on ∥ · ∥ ν .
Proof.
From the strong convexity of f s,a , for any x, y ∈ R ,
[Equation]
Then for q, q ′ ∈ R | SA | ,
[Equation]
since ∇ q E ν [ f s,a ( q ( s, a ))]
=
ν ◦ f ′ s,a ( q ) , which gives our result.
Lemma 8.
Suppose Assumption 3 holds.
Then for all ( q, w ) ∈ Q × W , w.p. ≥ 1 − δ ,
[Equation]
where ϵ qstat =
[Equation]
Proof.
From the linearity of the expectation, it is clear that L qf ( q, w ) =
E [ (cid:98) L qf ] .
Let l i =
w ( s i , a i )
( r ( s i , a i )
+ γq ( s ′
i , π ) − q ( s i , a i )) .
From Assumption 3,
[Equation]
Then using Hoeffding’s inequality with union bound, for all q, w ∈ Q × W , w.p. ≥ 1 − δ ,
[Equation]
n n
[Equation]
Lemma 9.
Under Assumptions 1, 2, 3, w.p. ≥ 1 − δ ,
[Equation]
where ϵ qstat is given in Lemma 8.
Proof.
We decompose the error as follows:
[Equation]
[Equation]
Combining the terms gives the result, and we provide a brief justification for each inequality below.
Terms (2) and (5) follow from Lemma 8.
Term (1) ≥ 0 since ( q π , w ∗ f ) is the saddlepoint solution.
Term (3) ≥ 0 , since (cid:98) L qf ( (cid:98) q, (cid:98) w ) =
(cid:98)
L qf ( (cid:98) q, (cid:98) w ( (cid:98) q )) , and (cid:98) q = arg min q ∈Q (cid:98)
L qf ( q, (cid:98) w ( q )) , and q π ∈ Q .
Term (4) ≥ 0 because w ∗ f ∈ W .
B.2. Proof of Lemma
Since strong duality holds, the saddle point ( q π , w ∗ f ) satisfies the KKT conditions.
Then from stationarity, for all ( s, a ) ,
[Equation]
Writing this in matrix form, letting f ′ ( q π ) be shorthand for [ f ′ s,a ( q π ( s, a ))] s,a ∈ R S×A , w ∗ f must satisfy the equality:
[Equation]
B.3. Proof of Proposition
Rearranging the closed form of w ∗ f from Lemma 3 and taking the absolute value of both sides,
[Equation]
Then dividing both sides by d D element-wise, this implies
[Equation]
As the above inequality holds for all ( s, a ) ,
[Equation]
C. Proofs for SectionC.1. Derivation of Lagrangian Objective (8)
For completeness, we demonstrate how the Lagrangian objective in (8) is derived from the constrained convex program in (7).
Letting q ∈ R S×A be the dual variable, (7) can be written in Lagrangian form and rearranged as follows:
[Equation] which is exactly (8) [Equation] C.2. Proof of Lemma
From the KKT stationarity conditions:
[Equation]
or in matrix form, letting f ′ ( w π ) be shorthand for [ f ′ s,a ( w π ( s, a ))] s,a ∈ R S×A ,
[Equation]
Then q ∗ f must satisfy
[Equation]
C.3. Proof of Theorem
The proof is of a similar nature as the proof of Theorem 2 (Appendix B.1).
From Assumption 4 and Lemma 7, we know that that L wf ( w, q ∗ f ) is an M -strongly convex function in w on the || · || 2 ,η norm.
Since ( w π , q ∗ f ) is the saddle point solution of
L wf [Equation] from strong convexity we know that the error of (cid:98) w is bounded as
[Equation]
Lemma 11)
[Equation]
where ϵ wstat is given in Lemma 10.
Remark 5.
In Theorem 6 of the main text, there is an additional O ( C wf / √ n ) term in the statistical error ϵ wstat , which would arise if the regularization function E η [ f s,a ( w ( s, a ))] were to be estimated from samples.
However, we state early on in the paper that we assume the regularizer can be calculated exactly, as sampling is a trivial extension.
Correspondingly, the correct expression for the statistical error is:
[Equation]
and, to remain consistent with the rest of the paper, we provide the proof and lemma for this ϵ wstat below.
Lemma 10.
Suppose Assumption 6 holds.
Then for all ( w, q ) ∈ W × Q , w.p. ≥ 1 − δ ,
[Equation]
where ϵ wstat =
(1 + γ ) C w W C
[Equation]
(cid:113)
[Equation]
n.
Proof.
Let l i =
w ( s i , a i )( γq ( s ′ i , π ) − q ( s i , a i )) .
Using Assumption 6, | l i | ≤ (1 + γ )
||
w || ∞ || q || ∞ ≤ (1 + γ ) C w W C w Q
Then using Hoeffding’s inequality with union bound, w.p. ≥ 1 − δ/
we have that for all w, q ∈ W × Q , (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)n n (cid:88)
i
[Equation]
[Equation]
n
Similarly, for all q ∈ Q , w.p
[Table]
Since L wf ( w, q ) =
E η [ f s,a ( w ( s, a ))]
+ E d D [ l i ] + E µ
[ q ( s 0 , π )] , but the first term can be calculated exactly, taking a union bound over the above two inequalities, we have that w.p. ≥ 1 − δ ,
[Equation]
Lemma 11.
Under Assumptions 4, 5, 6, w.p. ≥ 1 − δ ,
[Equation]
Proof of Lemma 11 We decompose the error as follows:
[Equation]
[Equation]
Combining the inequalities gives the result.
We give a brief justification for each term below.
Terms (1) and (4) follow from
Lemma 10.
Term (2) ≥ 0 , since q ∗ f ∈ Q .
Term (3) ≥ 0 since w π ∈ W and (cid:98) w = arg max w ∈W (cid:98) L wf ( w, (cid:98) q ( w )) .
Term (5) ≥ 0 since ( w π , q ∗ f ) is a saddle point solution.
D. Additional Details of the Experiments D.1. Derivation
We now derive the system of equations for our value function estimation experiments in Section 6.
Letting the regularization function be f s,a ( x ) = 12 x 2 for all ( s, a ) , the objective is
[Equation]
Letting E n denote the empirical average over D for clarity, with empirical samples and the linear classes Q , W , the objective becomes:
[Equation]
Since β ∈ R d , max w ∈W (cid:98) L qf ( q, w )
=
+ ∞ for any q , unless α sets the the second term to 0.
This is satisfied by α such that
[Equation]
However, there may in general be infinite feasible α depending on the linear features and samples.
For our specific linear parameterization of Q , W , the constraints form an underdetermined d × k system of equations, which has infinite solutions.
This is where the regularization term E ν [ α ⊤ ϕ ( s, a ) ϕ
( s, a )
⊤ α ] comes into play.
For any regularizing distribution ν , our method will output a solution that minimizes this term, i.e. that minimizes the norm of q =
Φ ⊤ α on ν .
If ν = 0 , for example, the algorithm will output any feasible point; if ν = 1 / |SA| , the algorithm will output q with smallest L2 norm.
Connection to LSTDQ When using the same linear class for W and Q , the solution to the constraints in (3) (i.e., ignoring the regularization objective)—if
the solution is unique given matrix invertibility—coincides with LSTDQ (Uehara et al., 2020).
As mentioned in Section 2, LSTDQ enjoys function-estimation guarantees under matrix invertibility.
In fact, we believe it is possible to extend the analysis even when Q and W use different features of dimensions d and k , respectively; as long as k ≥ d and the matrix in (3) has full row-rank 8 (i.e., overdetermined ), similar guarantees for LSTDQ should still hold, though we are not aware of an explicit documentation of this fact.
In contrast, our setup is more challenging as we are in the regime of k < d , and the constraints in (3) is underdetermined , nullifying the guarantees of LSTDQ.
In such cases, the use of regularization is important for guaranteeing function estimation, as also shown in our experiments.
D.2. Experimental Setup
Feature Design
In total, the tabular environment has 400 state-action values, and we design Φ to aggregate states that correspond to unique entries (within 3 decimal places) of q π .
In Figure 1, (cid:101) Φ is composed of the set of features given by
[Equation]
The first of these two entries is the closed-form solution of w ∗ f given in Lemma 3, and satisfies the realizability requirements of all methods; the second is included for optimization stability.
In Figure 2, we use a model with constant value equal to the average value of q π on the support of p , i.e. q = 1 / |SA| (cid:80) s,a q π ( s, a ) · 1 { p> 0 } .
To maintain realizability when the model is included in the regularization function, (cid:101) Φ is composed of the set (cid:110)
[Equation]
The reason why this preserves realizability is as follows.
When ν is the regularization distribution, and the input model is (cid:101) q
= ( mq π + [Equation] 1 − m ) q ) ◦ 1 ( p > 0))
for some constant q , the closed-form solution w ∗ f can be expanded as
[Equation]
which implies w ∗ f can be expressed as a linear combination of the three previously defined features.
Solver We solve the linear system using CVXPY with optimizer SCS (Diamond & Boyd, 2016; Agrawal et al., 2018).
Environment The Gridwalk is a 10x10 environment with 4 actions corresponding to cardinal directions.
The objective is to reach the goal state (lower right corner).
In each state, the agent receives a reward inversely proportional to its distance from a goal state.
Each trajectory terminates after 100 steps.
The initial states are randomly distributed over the upper half of the grid.
The target policy is defined to be a deterministic optimal policy that always moves towards the goal by first going right, and then down.
To create a strong shift, the behavioral policy is designed to largely explore only the bottom left portion of the grid, providing poor coverage over the target policy and starting states.
Specifically, letting the following probabilities refer to distributions over actions [RIGHT, DOWN, LEFT, UP], the target policy π has distribution [1 , 0 , 0 , 0] over actions until it hits the right wall, then [0 , 1 , 0 , 0] .
The behavior policy takes [0 . 1 , 0 . 4 , 0 . 5 , 0] until it hits the right wall, then takes [0 , 0 . 5 , 0 . 5 , 0] .
E. Approximation and Optimization Error
The main results of this paper (Theorems 2, 6) utilize assumptions on realizability (Assumption 2, 5), as well as (implicit) assumptions of perfect optimization.
In this section, we analyze how approximation errors, i.e. when the saddle point solution is not contained in Q × W , and optimization errors affect our error bounds.
Due to the similarity in proofs between value function and weight learning, we provide them only for value function learning; analogous methods can be used to derive similar results for weight learning.
E.1. Finite-sample Guarantees
First, we relax the realizability requirements of Assumption 2.
Define the approximation errors:
[Equation]
ϵ approx,q is composed of the worst-case weighted combination of Bellman errors of the best candidate q ∈ Q , as well as the difference between the regularization function at q and q π .
The error ϵ approx,w measures the distance between the best candidate w ∈ W and the saddle point solution w ∗ f by projecting the difference onto the worst-case Bellman error T π q − q .
Remark 6.
To increase intuition of ϵ approx,q , we can relax the difference in regularization terms as E ν [ f s,a ( q ( s, a ))
− f s,a ( q π ( s, a ))] ≤ C qf ′ ||
q π − q || 2 ,ν , which is also the norm upon which the (cid:98) q estimation guarantee is given (Theorem 2) .
Reflecting the nature of the value function estimation task, this states that, even if there is a candidate q ∈ Q with low
Bellman error (e.g. if data is sparse), ϵ approx,q will still be large if q is far from q π on the desired distribution ν .
Next, we can also relax the (implicit) assumptions that the estimates ( (cid:98) q, (cid:98) w ) are the true optima of (5), e.g. (cid:98)
q = arg min q ∈Q (cid:98)
L qf ( q, (cid:98) w ( q )) and (cid:98) w = arg max w ∈W
[Equation]
L qf ( (cid:98) q, w ) .
Letting (cid:98) w ( q ) =
arg max w ∈W (cid:98) L ( w, q ) , define the following optimization errors:
[Equation]
ϵ opt,w states that the estimate (cid:98) w should not be too far from the best discriminator in W for (cid:98) q , while ϵ opt,q states that the estimate (cid:98) q should not be too far from the minimax solution.
Using the above definitions, we provide the following generalization of Theorem 2, which accounts for approximation and optimization errors.
Theorem 12.
Under Assumptions 1 and 3, with probability at least 1 − δ ,
[Equation]
where ϵ qstat is given in Theorem 2.
E.2. Proof of Theorem
The proof takes the same overall steps as the proof of Theorem 2 (Appendix B.1), but relies on Lemma 13 to incorporate the approximation and optimization errors:
[Equation]
Below, we state and prove the helper lemma, which bounds the difference between the Lagrangian objective (4) at the saddle point ( q π , w ∗ f ) and the point ( (cid:98) q, w ∗ f ) :
Lemma 13.
Under Assumptions 1 and 3, w.p. ≥ 1 − δ ,
[Equation]
Proof.
With some abuse of notation (as (cid:101) q [Equation] (cid:101) w previously referred to models used with the regularizer) [Equation]
for brevity in this section
[Equation]
let (cid:101) q be the minimizer of ϵ approx,q and (cid:101) w be the minimizer of ϵ approx,w .
That is,
[Equation]
Decompose the error as follows:
[Equation]
First, (1) holds because ( q π , w ∗ f ) is the saddle point solution of L qf over all q, w ∈ R × R .
The statistical errors in (3) and (6) follow from Lemma 8.
Next, we justify the optimization errors.
For (4),
[Equation]
For (5),
[Equation]
Finally, we justify the approximation errors, starting with (2).
Note that for any q, w ∈ Q × W ,
[Equation]
Then since (cid:101) q was chosen to minimize the above expression,
[Equation]
Next we justify (8).
For any w ∈ W and q ∈ Q ,
[Equation]
Then since (cid:101) w was chosen to minimize the RHS of the above inequality,
[Equation]
Combining these inequalities gives the lemma statement.
F. Off-Policy Return Estimation
Section 4 demonstrates how q-value estimates (cid:98) q can be obtained [Equation] and Section 5 demonstrates how weight estimates (cid:98) w can be obtained.
The estimates (cid:98) q
and/or
(cid:98) w can additionally be used for downstream off-policy evaluation (OPE) of the policy
’s value J ( π ) , which can be equivalently defined in the following three ways:
[Equation]
With finite samples and estimates (cid:98) q and (cid:98) w approximating q π and w π , respectively, their corresponding off-policy estimators are:
[Equation]
While the OPE estimator (cid:98) J dr ( π ) utilizes both the weights and value functions, (cid:98) J w ( π ) and (cid:98) J q ( π ) utilize only one or the other.
As a result, when (cid:98) q and (cid:98) w are estimated as in Sections 4 and 5, respectively, (cid:98) J w ( π ) and (cid:98) J q ( π ) both inherit their
O ( n − 1 / 4 ) sample complexities:
Corollary 14.
Suppose Assumptions 1, 2, and 3 hold, and let ( (cid:98) q, )
[Equation]
( q, w ) .
Then with probability ≥ 1 − 2 δ ,
[Equation]
where ϵ q eval = (1 − γ )
C q Q (cid:113) 2log 2 |Q| δ / n 0 , C µ π 0 /ν
=
||
µ π 0 / ν || ∞ , and ϵ qstat is as in Theorem 2.
Corollary 15.
Suppose Assumption 4, 5, and 6 hold, and let [Equation]
= arg min w ∈W arg max q ∈Q (cid:98) L wf ( q, w ) .
Then with probability ≥ 1 − 2 δ ,
[Equation]
where ϵ weval =
C w W (cid:113) 2log 2 |W| δ n , C d D /η =
∥
d D / η ∥ ∞ , and ϵ west is as in Theorem 6.
However, when (cid:98) q and (cid:98) w are used together in the doubly robust estimator (cid:98) J dr , their estimation error becomes multiplicative, and (cid:98) J dr ( π ) can achieve the O ( n − 12 ) fast rate of convergence.
In Theorem 16 below, we present two versions this guarantee.
The first requires no additional assumptions beyond d D > 0 , which we already make (see footnote 5), but involves the largest singular value of I − γP π , which may be difficult to characterize.
The second utilizes an additional assumption, and replaces the singular value with an occupancy ratio, stated below.
The assumption requires that all next states s ′ are also present as states s in transitions of d D (a condition which may reasonably hold in practice), and is also made by (Uehara et al., 2021).
Assumption 7 (Next State Coverage) .
Let d D ( s )
= (cid:80) a d D ( s, a ) be the marginal distribution of states s in d D , and
d Ds ′ ( s ) := (cid:80) s ′ ,a ′ P ( s | s ′ , a ′ )
d D ( s ′ , a ′ ) be the marginal distribution of next states s ′ .
Suppose
[Equation]
Theorem 16.
Suppose Assumption 1, 2, 3, 4, 5, and 6 hold.
Let (cid:98) w and (cid:98) q be estimated from:
[Equation]
Then with probability ≥ 1 − 3 δ ,
[Equation]
If Assumption 7 additionally holds, with probability ≥ 1 − 3 δ ,
[Equation]
where ϵ dreval =
(1 − γ ) C q Q (cid:113) 2log 2 |Q| δ / n 0 + C w W (1 + (1 + γ ) C q Q ) (cid:113) 2log 2 |W||Q| δ / n , σ max denotes the largest singular value, and ϵ qest and ϵ west are as in Theorems 2 and 6.
As the evaluation error ϵ dreval in Theorem 16 is O ( n − 1 / 2 ) , the sample complexity of doubly robust estimation is rate-limited by ϵ west · ϵ qest , the product of weight and value function estimation errors.
If both functions can be estimated at an O ( n − 1 / 4 ) rate, as is true of our method, then (cid:98) J dr ( π ) attains the overall O ( n − 1 / 2 ) fast rate.
Finally, while Theorem 16 assumes for simplicity that the same Q , W classes are used in both of its optimization problems, it can easily be extended to the case where different pairs of function classes are used as long as the required assumptions hold.
Remark 7 (Comparison to Related Work) .
(Yang et al., 2020) conduct experiments comparing off-policy evaluation using (cid:98) J q ( π ) , (cid:98) J w ( π ) , (cid:98) J dr ( π ) , and generally observe that (cid:98) J dr ( π ) has higher variance and worse performance than either (cid:98) J q ( π ) or (cid:98)
J w ( π ) .
Though at first glance this may appear to contradict Theorem 16, that is actually not the case; in fact, our theoretical analysis provides insight into why (Yang et al., 2020) may observe such a phenomenon.
In contrast to
Theorem 16, when using (cid:98) J dr ( π ) (Yang et al., 2020) utilize saddle point predictions ( (cid:98) q, (cid:98) w ) from either only value function learning or only weight learning, e.g. ( (cid:98) q, (cid:98) w ) =
arg min q ∈Q arg max w ∈W (cid:98)
L qf ( q, w ) that approximates ( q π , w ∗ f ) .
Continuing with this example (and the same applies to weight learning), it is clear from our analysis that (cid:98) w estimated in such a manner may not approximate w π at all, leading to increased estimation error of (cid:98) J dr ( π ) over (cid:98) J q ( π ) .
First, the closed-form solution we have derived for w ∗ f in (Lemma 3) shows that w ∗ f may have a significantly different magnitude from w π .
Second, even if ν and f were chosen such that w ∗ f ≈ w π , as per the reasons stated in Section 4.1, we are not even guaranteed to output (cid:98) w close to w ∗ f since L qf is not regularized in w .
In order to obtain the estimation benefits of doubly robust estimation, our analysis shows that (cid:98) q and (cid:98) w should be separately estimated from their respective optimization problems, then combined in (cid:98) J dr ( π ) .
This is in accordance with similar results from Kallus & Uehara (2020) and Uehara et al.
F.1. Proof of Corollary
Let (cid:101) J
( π ) =
(1 − γ ) E {+µ 
+}
[ (cid:98) q
( s, π )] .
We decompose the error as
[Equation]
First we bound | (cid:98) J ( π ) − (cid:101)
J ( π ) | .
Using Hoeffding’s with union bound, for all q ∈ Q , w.p. ≥ 1 − δ , (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)nn (cid:88) i =
[Equation]
which implies | (cid:98) J ( π ) − (cid:101) J ( π ) | ≤ ϵ qeval .
For the second term, let C µ π 0 /ν
=
||
µ π 0 /ν || ∞ .
Then w.p. ≥ 1 − δ
[Equation]
using Theorem 2 in the last line.
F.2. Proof of Corollary
Let (cid:101) J
( π ) =
E d D [ (cid:98) w ( s, a ) r ( s, a )] .
We decompose the error as
[Equation]
For the first term, using Hoeffding’s with union bound, w.p. ≥ 1 − δ , for all w ∈ W ,
[Equation]
n
[Equation]
i
[Equation] which implies | [Equation] ) | ≤ ϵ weval .
For the second term,
[Equation]
w.p. ≥ 1 − δ , using Theorem 6 in the last line.
Taking a union bound over both terms gives the stated result.
F.3. Proof of Theorem
Let (cid:101) J
( π ) =
[Equation]
1 −
[Equation]
) E
[Equation]
[ (cid:98) q
[Equation]
s, a )] + E d D [ (cid:98) w ( s, a )( r + (cid:98) q ( s ′ , π ) − (cid:98) q ( s, a ))] .
Again we decompose the error as:
[Equation]
|
[Equation]
|
[Equation]
J [Equation] ) − J ( π ) | . For the first term, since E [ (cid:98) J dr ( π )] = (cid:101) J ( π ) , w.p. ≥ 1 − δ we have that ∀ q, w ∈ Q × W , | (cid:98) J dr
[Equation]
[Equation]
[Equation]
) |
[Equation]
1 − [Equation] ) C q Q [Equation]
log
[Equation]
δ n
[Equation]
(cid:115)
[Equation]
| δ n :=
ϵ dreval
For the second term,
[Equation]
where the last equality is due to the fact that (1 − γ ) µ π
= ( I − γP π )( d D · w π ) , and the final inequality is from Cauchy-Schwarz.
We can automatically bound the || (cid:98) w − w π
|| 2 ,d D term using Theorem 6, and it remains to bound ||
( I − γP π )( (cid:98) q − q π )
|| 2 ,d D .
We will consider two cases, first when d D > 0 thus Diag ( d D ) is invertible, and second, when Assumption 7 is satisfied.
In the first case, let D =
Diag ( d D ) , which by assumption is invertible.
Then
[Equation]
in the last line using the fact that the eigenvalues of a matrix A and L − 1 AL are the same for any invertible matrix L .
Thus, denoting the largest singular value of a matrix by σ max ,
[Equation]
Using Theorem 6 and Theorem 2 in the last line to control the errors of (cid:98) w and (cid:98) q in the last line, followed by a union bound over the three inequalities, gives the result.
For the second case, we can directly apply Lemma 17:
[Equation]
and again applying Theorem 6 and Theorem 2 gives the result.
Lemma 17 uses Assumption 7 to bound the distance in value functions under the transition operator, and is stated and proved below.
Lemma 17.
Under Assumption 7,
[Equation]
Proof.
Define ||
P
π
|| 2 ,d D := sup x ̸ =
||
P π x || 2 ,d D / || x
|| 2 ,d D .
Then || P
[Equation]
q − q π
[Equation]
|| 2 ,d D ≤ || P
π
||
[Equation]
d D
[Equation]
|
[Equation] q [Equation] q
[Equation]
||
[Equation]
d D .
It remains to bound ||
P
π
|| 2 ,d D .
For any x ,
[Equation]
This implies that ||
P π || 2 ,d D ≤ (cid:112) C s ′ /s
C π/π D [Equation] which gives the stated result
[Equation]