Offline Policy Evaluation for Reinforcement Learning with Adaptively Collected Data
Abstract
Offline RL is an important step towards making data-hungry RL algorithms more widely usable in the real world, but conventional assumptions on the distribution of logging data do not apply in some key real-world scenarios.
In particular, it is unrealistic to assume that RL practitioners will have access to sets of trajectories that simultaneously are mutually independent and explore well.
We propose two natural ways to relax these assumptions: by allowing the data to be distributed according to different logging policies independently and by allowing logging policies to depend on past trajectories.
We discuss Offline Policy Evaluation (OPE) in these settings, analyzing the performance of a model-based OPE estimator when the MDP is tabular.
Introduction
Reinforcment Learning (RL) studies algorithms that learn to optimize a reward criterion by dynamically interacting with an environment. By balancing exploration with exploitation, online RL algorithms can very effectively learn to behave optimally.
However, this mode of randomized exploration precludes some crucial use-cases for RL, such as for running medical trials, or training self-driving cars.
Even in (comparatively) low-stakes applications like advertising, adopting elementary RL algorithms would mean throwing away vast reserves of data gathered from previous strategies.
Both of these problems would be partially addressed by practical methods for offline learning, which seeks to solve the control problem using existing data.
In the offline RL setting, the goal is to perform RL tasks using existing data, D , generated by some logging policy, µ , and MDP M .
In Offline Policy Evaluation (OPE), we seek to estimate the value of a target policy π under M .
In Offline Learning (OL), the goal is to use D to find a good policy π ∈ Π where Π is some policy class.
Practically, OL would allow RL practitioners to exercise more control over the gathering of training data.
However, in order for D to be a rich enough dataset to learn from, strong assumptions need to be made about the exploratory properties of µ .
This is in addition to the already strong assumption that the set of trajectories is i.i.d.
In Section 3, we discuss the problems that such assumptions pose and propose ways of weakening them.
We believe the most fruitful of these to be Adaptive OPE (AOPE), where we allow each trajectory to be distributed according to a different logging policy, which may depend on previous data.
Some scenarios that AOPE covers and OPE does not are presented below.
The dataset D has been collected over a long period of time, during which unrecorded changes have been made to the policy.
An example of this would be the portfolio of a longterm client at an investment company, which will certainly change over time.
The dataset D was gathered by humans, and therefore influenced by a number of unobserved factors.
For example, a doctor prescribing medicine may make a determination based on her conversation with the patient – a factor not recorded in any state variable. The dataset D has been generated through a combination of algorithmic decisions and human input. For example, an engineer may tune the parameters of a recommendation algorithm based on the observed level of engagement. If it can be demonstrated that existing OPE estimators extend to the AOPE setting with no performance drop, a new class of use-cases for Offline RL emerges.
If AOPE is, in fact, verifiably harder than
OPE, practitioners will know that to achieve optimal performance in Offline RL tasks, they will need to collect remarkably clean data.
This paper represents our first steps towards resolving this key question.
Related Work
The OPE literature is vast.
We do not attempt to provide a survey of the excellent body of work but instead refer readers to the recent work of Mou et al.
[2022] and the references therein for a more comprehensive discussion.
To the best of our knowledge, we are the first to study the problem of OPE under the adaptive data setting.
Most existing work on OPE that we have seen makes the assumption that the data are collected iid from a single logging policy.
The only exception is the work of Kallus et al.
[2020] who studied OPE from multiple loggers, but they only considered the contextual bandits model and non-adaptive loggers, while we studied the RL with possibly adaptively chosen loggers.
OPE is also closely related to the average treatment effect estimation problem in the causal inference literature, but typically only only one-step decision is considered and the observational data are assumed to be iid.
Let ∆( X ) be the set of all PMFs over X , for |X| < ∞ .
Let [ H ] := { 1 , ..., H }
A Tabular, Finite-Horizon Markov Decision Process is a tuple ( S , A , r, P, d 1 , H ) , where S is the state space ( |S| =: S ), and A is the action space ( |A|
=: A ).
Its dynamics are governed by a nonstationary transition kernel, P = { P h : S ×A → ∆( S ) } Hh =1 , where P h ( s ′ | s, a ) is the probability of transitioning to state s ′ ∈ S after taking action a ∈ A from state s ∈ S at time h ∈
[ H ] .
r is a collection of reward functions { r h : S ×
A → [ − 1 , 1] } Hh =1 .
Finally, d 1 ∈ ∆( S ) is the initial state distribution of the MDP and H is the horizon.
A policy, π , is a collection of maps, { π h : S → ∆( A ) } Hh =1 .
Running a policy on an MDP will yield a trajectory τ i ∈ ( S ×
A × [ − 1 , 1]) H .
Together, the policy and MDP induce a distribution over trajectories, as well as a Markov Chain with transitions notated as P πh ( s ′ | s ) :
= (cid:80) a P h ( s ′ | s, a ) π h ( a | s ) .
In a set of trajectories { τ i } ni =1 , we define n h,s,a to be the number of visitations to ( s, a ) at timestep h .
v π :=
E π [ (cid:80) Hi =1 r
i | s 1 ∼ d 1 ] is the value of the policy π , where the expectation is over the π -induced distribution over trajectories.
Similarly, V π ( s ) := E π [ (cid:80) Hi =1 r i | s 1 = s ] . d πh ( s, a ) is defined to be the probability of ( s h , a h ) occurring at time step h in a trajectory distributed according to π .
Problem Formulation and Motivation
Naturally, both OPE and OL are hopeless if the logging-policy does not explore well.
If the logger, µ , does not visit a state that the target policy, π , visits very often, we will not be able to form an accurate estimate of the target value v π when doing OPE.
In OL, this issue is compounded by the fact that missing any state may correspond to missing high-value outcomes.
Thus, upper bounds on the performance of OPE or OL algorithms are given in terms of an exploration parameter, like
[Equation]
We see that, in the adaptive regime, each policy depends on all previous trajectories.
This induces dependence between the trajectories. Estimators that closely match lower bounds on relevant metrics have been established in existing work, such as Duan et al.
However, the practicability of such bounds has been challenged.
Xiao et al.
[2022] point out that it is difficult in practice to find a logging policy with a reasonable exploration parameter.
In what they consider a more realistic, “tabula rasa" case (where the logging policy is chosen without knowledge of the MDP), they show a sample complexity exponential in H and S to be necessary in offline learning.
Besides this limitation, we note that the motivating application of learning from existing, humangenerated data remains unfulfilled in the conventional OPE setting.
We would not expect these data to be identically distributed, or even independent (as the data collected in trajectory j almost certainly influences future policies µ j +1 , ... ).
Thus motivated, we augment our formulation of the OPE problem to more realistically accommodate intelligent choices of logging-policy.
We consider this to be middle ground between the sanguine assumptions on d m common to Yin and Wang [2020], Yin et al.
[2021], Duan et al. [2020], and the assumption of total ignorance found in Xiao et al.
To this end, this paper studies two settings:
OPE on D = {
τ i ∼ µ i } ni =1 , where µ 1 , ...µ n are distinct policies chosen in advance, and the trajectories unfold independently.
We call this the Non-Adaptive OPE (NOPE).
OPE on D = {
τ i ∼ µ i } ni =1 , where µ 1 , ...µ n are chosen adaptively.
That is, µ i may depend on the trajectories τ 1 , ...
τ i − 1 .
NOPE allows us to hedge our bets with respect to the logging policy we choose.
In particular, when we derive a bound on the MSE of the model-based estimator from [Yin and Wang, 2020], it will be in terms of the minimum of the average state-action occupancy:
[Equation]
If there is some preexisting knowledge on the MDP, M , it may be easier to propose n logging policies with (2) bounded away from zero than a single logging policy with (1) bounded away from zero.
NOPE does not adequately address concerns over finding good logging policies, but rather dilutes the problem to that of finding a suite of policies that is good on average.
Furthermore, it only eliminates assumptions on identical distribution, and sets aside issues raised earlier over dependence between trajectories.
Both of these problems are better addressed by AOPE.
When logging policies can be tuned according to previous trajectories, there is scope for starting from “tabula rasa”, and iteratively refining the logging policy as we learn about the MDP.
In other words, the logger can leverage online exploration techniques.
Furthermore, by allowing arbitrary statistical dependence on previous trajectories, AOPE addresses the key scenario of learning from intradependent, manually-collected datasets. We conclude this section by noting that minimax bounds from the non-adaptive OPE setting can be easily recovered in the following manner. For any dataset, D , let N :=
min h,s,a n h,s,a be the number of occurrences of the least-observed ( s h , a h ) pair. If we consider a revised dataset, D ′ , that keeps only the first N transitions out of ( s h , a h ) for all s h , a h , we see that all transitions are now independent conditioned on N . Thus, under the conventional assumptions on exploration-parameters, the problem reduces to a generative-model type setting. In particular, Theorem 3.7 in Yin et al.
[2021] implies minimax-optimal offline learning for the adaptive case.
However, we do not like throwing data away in this manner, and conjecture that it should not be necessary to do so.
It would be preferable to obtain bounds that adapt to the quantities { n h,s,a } h,s,a .
To this end, we explore the extent to which instance-dependent bounds on estimation error can be recovered in the adaptive setting.
Method
We consider the estimator of v π studied in [Yin and Wang, 2020].
This boils down to computing the value of a policy under the approximate MDP defined by ( S , A , ˆ P, ˆ r , ˆ d 1 ), with the estimators ˆ P , ˆ r and ˆ d 1 defined below.
That is, if D = { τ 1 , ...τ n } , and τ i
= ( s i 1 , a i 1 , r i 1 , ...s
iH , a iH , r iH ) , we use plug-in estimates ˆ P h ( s ′ | s, a ) = {+n h,s,a,s ′ n h,s,a = n h,s,a (cid:88) i 1 { s ih = s,a ih = a,s ih +
 =+}
s ′ } ˆ r h ( s, a )
=
n h,s,a n (cid:88) k
= r kh 1 { s kh = s,a kh = a } subject to these quantities being well-defined ( n h,s,a ̸ = 0 ).
If n h,s,a = 0 , we can define them to be 0 .
We also define ˆ d 1 :=:
ˆ d π 1 := 1 n (cid:80) ni
=
e s i 1 to be the plug-in estimate of d 1 computed from D (where e j is the j th standard basis vector in R S ).
We then let:
[Equation]
and iteratively compute ˆ d πh :
=
ˆ P πh ˆ d πh − 1 for h = 1 , ...H .
Finally, we form the estimate
[Equation]
Current Results
We warm up by generalizing Yin and Wang [2020]’s bound on the MSE of ˆ v π to NOPE. By following the proof of Theorem 1 in Yin and Wang [2020], and making some mild modifications, we recover a bound of the MSE of ˆ v π in the Non-Adaptive OPE setting.
Theorem 1.
[MSE performance of ˆ v π in NOPE setting]
Suppose D is a dataset conforming to NOPE, and ˆ v π is formed using this dataset.
Let ¯ d m be as defined in (2).
Let τ s = max h,s,a [Equation] s,a [Equation]
n
[Equation]
i d µih ( s,a ) .
Let τ a = max h,s,a π h ( a | s )n (cid:80) i µ ih ( a | s ) . Then if n > 16log n ¯ d m and n >
[Equation]
1 n [Equation]
i d µih ( s ) } we have:
[Equation]
As a corollary, consider a “quasi-adaptive” data collection process, where each logging-policy, µ i , is run twice, generating i.i.d.
τ i and τ ′ i .
Suppose future logging-policies µ j>i are chosen by some algorithm E depending on τ
i but not τ ′ i .
We can use the same ˆ v π -estimator to perform OPE with
D shadow = { τ ′ i } , as long as the estimator doesn’t touch D
= { τ i } .
If we assume that average exploration is sufficient w.h.p.
over the execution of E , we can the bound MSE in this quasi-adaptive case using Theorem 1, the fact that the { τ ′ i } are mutually independent conditioned on { µ i } , and the tower rule.
Corollary 2.
Let E be the algorithm described in the paragraph above.
Assume that with high probability ( ≥ 1 − δ ), the policies µ 1 , ...µ n generated by E satisfy 1 N (cid:80) i
d µ i h ( s, a ) ≥ ¯ d m for all s ∈ S and a ∈ A , and for some ¯ d m .
Then
[Equation]
where ( ∗ ) is the bound on the MSE of the estimator in the non-adaptive case from Theorem 1.
We now turn our attention towards quantifying ˆ v π ’s performance on AOPE.
We first describe a high-probability, uniform error bound in terms of the number of visitations to each ( s h , a h ) tuple.
Theorem 3 (High-probability uniform bound on estimation error in AOPE) .
Suppose D is a dataset conforming to AOPE, and ˆ v π is formed using this dataset.
Then, with probability at least 1 − δ , the following holds for all policies π .
[Equation]
where n h,s,a is the number of occurrences of ( s h , a h ) in D , and K is an absolute constant.
This translates to the following worst-case bound, which underperforms the minimax-optimal bound (over deterministic policies) implied by Yin et al.
[2021] by a factor of √ H .
Corollary 4 (High-probability uniform bound on estimation error in AOPE) .
Suppose that D , ˆ v π are as in Theorem 3.
Then with probability 1 − δ , we have that
[Equation]
The proof of Theorem 3 follows by a simulation lemma-type expansion of the error, which leads to a dominant term of the form (cid:80)
h
E s h ,a h ∼ π, M
[( ˆ P h +1 ( ·| s h , a h ) − P
h +1 ( ·| s h , a h ))
T ˆ V πh +1 ] , and smaller terms governed by ˆ r and ˆ d 1 .
In order to get around the issue of dependence between trajectories, we cover all possible number of occurrences of each ( s h , a h ) across trajectories while applying concentration, leading to to the HSAn term inside the logarithm.
We also give a high-probability, instance-dependent, pointwise bound, which is suboptimal by a factor of √ H when translated into a worst-case bound.
In the pointwise case, we are able to shave off a √ S in the asymptotically dominant term.
Theorem 5 (Instance-dependent pointwise bound on estimation error in AOPE) .
Fix a policy π , suppose D is a dataset conforming to AOPE, and ˆ v π is formed using this dataset.
Assume that with high probability, n h,s,a ≥ n ¯ d m for all h, s, a for some ¯ d m > 0 .
Then with probability at least 1 − δ , we have: [Equation]
The above translates into the following worst-case bound.
Corollary 6 (Worst-case pointwise bound on estimation error in AOPE) .
Fix a policy π , suppose D is a dataset conforming to AOPE, and ˆ v π is formed using this dataset.
Then with probability at least 1 − δ , we have:
[Equation]
Inspired by Azar et al.
[2017], Theorems 5 is proved by applying concentration inequalities (with the same covering trick as Theorem 3) to ( ˆ P h
+1 − P h +1 )
V πh +
and ( ˆ P h +1 − P h +1 )
( ˆ V πh +1 − V πh +1 ) separately, instead of ( ˆ P h
+1 − P h +1 )
ˆ V πh +1 .
In order to treat the dominant term, we use Bernstein’s inequality. To recover the worst-case bound in the corollary, we analyze the variance term with the canonical equality (cid:80) h E π [Var s ′ ∼ P ( ·| s,a )
[ V πh ( s ′ )]] =
Var π
[ (cid:80) h r h ] − (cid:80) h
E π [Var[ E [ r h + V
πh +1 ( s ′ ) | s, a ]]] = O ( H 2 ) .  Numerical Simulations
In Section 4, we stated our conjecture that learning from adaptively collected datasets should not be harder than learning from mutually independent datasets. In this section, we present the early results of simulations designed to study this question. The simulations were conducted on a toy MDP with two states, two actions, and nonstationary dynamics, and performed as follows. Fix a type of adaptive algorithm, Z (we used the well-known exploration algorithms UCB-VI [Azar et al., 2017] and SARSA), and a target policy, π .
• Collect a dataset of k trajectories, D , by running Z .
Record the policies Z uses in Π Z • Collect a "shadow" dataset, D shadow by running each of the k policies in Π Z • For both datasets, and for a range of n ∈ { 1 , ...k } , estimate and ˆ v π using the first n trajectories in D , and do the same for D shadow
At a glance, Figure 2 suggests that the adaptive dataset does not suffer a performance hit in either adaptivity regime.
However, this data is noisy, and we intend to gather more extensive empirical evidence for this conclusion in future.
Next Steps
On the theoretical side, the instance-dependent results from Theorems 3 and 6 fail to recover the correct minimax behavior. This is because we could not salvage the martingale structure leveraged across timesteps in [Yin et al., 2021]. However, based on our early simulations and our intuition that throwing away data should not help us learn, we believe this gap to be an artifact of our analysis.
As future work, we intend to recover better bounds by more carefully analyzing the structure of the MDP, or else demonstrate (by means of a lower bound) that the estimator ˆ v π is worse at AOPE than OPE. On the experimental side, we plan on carrying out more detailed simulations than those presented in Section 7.
Empirically observing the behavior of certain error metrics under various adaptivity regimes may illuminate theoretical aspects of the problem (especially concerning how adaptivity can give rise to estimation bias for certain policies, an extension from Multiarm Bandits to RL of the phenomena studied in [Shin et al., 2019]).
We believe that theoretically clarifying the extent to which OPE methods carry over to the AOPE setting is a step towards making offline RL a yet more convincing candidate for real-world decision problems.