P ESSIMISM IN THE F ACE OF C ONFOUNDERS : P ROVABLY E FFICIENT O FFLINE R EINFORCEMENT L EARNING IN P ARTIALLY O BSERVABLE M ARKOV D ECISION P ROCESSES
Anonymous authors
Paper under double-blind review
A BSTRACT
We study offline reinforcement learning (RL) in partially observable Markov decision processes.
In particular, we aim to learn an optimal policy from a dataset collected by a behavior policy which possibly depends on the latent state.
Such a dataset is confounded in the sense that the latent state simultaneously affects the action and the observation, which is prohibitive for existing offline RL algorithms.
To this end, we propose the Proxy variable Pessimistic Policy Optimization ( P3O ) algorithm, which addresses the confounding bias and the distributional shift between the optimal and behavior policies in the context of general function approximation.
At the core of P3O is a coupled sequence of pessimistic confidence regions constructed via proximal causal inference, which is formulated as minimax estimation.
Under a partial coverage assumption on the confounded dataset, we prove that P3O achieves a n − 1 / 2 -suboptimality, where n is the number of trajectories in the dataset.
To our best knowledge, P3O is the first provably efficient offline
RL algorithm for POMDPs with a confounded dataset.
I NTRODUCTION
Offline reinforcement learning (RL) (Sutton and Barto, 2018) aims to learn an optimal policy of a sequential decision making problem purely from an offline dataset collected a priori, without any further interactions with the environment.
Offline RL is particularly pertinent to applications in critical domains such as precision medicine (Gottesman et al., 2019) and autonomous driving (Shalev-Shwartz et al., 2016).
In particular, in these scenarios, interacting with the environment via online experiments might be risky, slow, or even possibly unethical.
But oftentimes offline datasets consisting of past interactions, e.g., treatment records for precision medicine (Chakraborty and Moodie, 2013; Chakraborty and Murphy, 2014) and human driving data for autonomous driving (Sun et al., 2020), are adequately available.
As a result, offline RL has attracted substantial research interest recently (Levine et al., 2020).
Most of the existing works on offline RL develop algorithms and theory on the model of Markov decision processes (MDPs).
However, in many real-world applications, due to certain privacy concerns or limitations of the sensor apparatus, the states of the environment cannot be directly stored in the offline datasets.
Instead, only partial observations generated from the states of the environments are stored (Dulac-Arnold et al., 2021).
For example, in precision medicine, a physician’s treatment might consciously or subconsciously depend on the patient’s mood and socioeconomic status (Zhang and Bareinboim, 2016), which are not recorded in the data due to privacy concerns.
As another example, in autonomous driving, a human driver makes decisions based on multimodal information of the environment that is not limited to visual and auditory inputs, but only observations captured by
LIDARs and cameras are stored in the datasets (Sun et al., 2020).
In light of the partial observations in the datasets, these situations are better modeled as partially observable Markov decision processes (POMDPs) (Lovejoy, 1991).
Existing offline RL methods for MDPs, which fail to handle partial observations, are thus not applicable.
In this work, we make the initial step towards studying offline RL in POMDPs where the datasets only contain partial observations of the states.
In particular, motivated from the aforementioned real-world applications, we consider the case where the behavior policy takes actions based on the states of the environment, which are not part of the dataset and thus are latent variables.
Instead, the trajectories in datasets consist of partial observations emitted from the latent states, as well as the actions and rewards.
For such a dataset, our goal is to learn an optimal policy in the context of general function approximation.
Furthermore, offline RL in POMDP suffers from several challenges.
First of all, it is known that both planning and estimation in POMDPs are intractable in the worst case (Papadimitriou and Tsitsiklis, 1987; Burago et al., 1996; Goldsmith and Mundhenk, 1998; Mundhenk et al., 2000; Vlassis et al., 2012).
Thus, we have to identify a set of sufficient conditions that warrants efficient offline RL.
More importantly, our problem faces the unique challenge of the confounding issue caused by the latent states, which does not appear in either online and offline MDPs or online POMDPs.
In particular, both the actions and observations in the offline dataset depend on the unobserved latent states, and thus are confounded (Pearl, 2009).
[Figure]
The dotted nodes indicate that the variables are not stored in the offline dataset.
Solid arrows indicate the dependency among the variables.
In specific, the action A h is specified by the behavior policy which is a function of S h ( blue arrows).
Moreover, both the observation O h and reward R h depend on the state S h ( red arrows).
We remark that S h affects both A h and ( O h , R h ) and thus serves as an unobserved confounder .
Such a confounding issue is illustrated by a causal graph in Figure 1.
As a result, directly applying offline RL methods for MDPs will nevertheless incur a considerable confounding bias.
Besides, since the latent states evolve according to the Markov transition kernel, the causal structure is thus dynamic, which makes the confounding issue more challenging to handle than that in static causal problems.
Furthermore, apart from the confounding issue, since we aim to learn the optimal policy, our algorithm also needs to handle the distributional shift between the trajectories induced by the behavior policy and the family of target policies.
Finally, to handle large observation spaces, we need to employ powerful function approximators.
As a result, the coupled challenges due to (i) the confounding bias, (ii) distributional shift, and (iii) large observation spaces that are distinctive in our problem necessitates new algorithm design and theory.
To this end, by leveraging tools from proximal causal inference (Lipsitch et al., 2010; Tchetgen et al., 2020; Miao et al., 2018b;a), we propose the Proxy variable Pessimistic Policy Optimization ( P3O ) algorithm, which provably addresses the challenge of the confounding bias and the distributional shift in the context of general function approximation.
Specifically, we focus on a benign class of POMDPs where the causal structure involving latent states can be captured by the past and current observations, which serves as the negative control action and outcome respectively (Miao et al., 2018a;b; Cui et al., 2020; Singh, 2020;
Kallus et al., 2021; Bennett and Kallus, 2021; Shi et al., 2021).
Then the value of each policy can be identified by a set of confounding bridge functions corresponding to that policy, which satisfy a sequence of backward moment equations that are similar to the celebrated Bellman equations in classical RL (Bellman and Kalaba, 1965).
Thus, by estimating these confounding bridge functions from offline data, we can estimate the value of each policy without incurring the confounding bias.
More concretely, P3O involves two components — policy evaluation via minimax estimation and policy optimization via pessimism.
Specifically, to tackle the distributional shift, P3O returns the policy that maximizes pessimistic estimates of the values obtained by policy evaluation.
Meanwhile, in policy evaluation, to ensure pessimism, we construct a coupled sequence of confidence regions for the confounding bridge functions via minimax estimation, using function approximators.
Furthermore, under a partial coverage assumption on the confounded dataset, we prove that P3O achieves a (cid:101)
O ( H (cid:112) log( N fun ) /n ) suboptimality, where n is the number of trajectories, H is the length of each trajectory, N fun stands for the complexity of the employed function classes (e.g., the covering number), and (cid:101) O ( · ) hides logarithmic factors.
When specified to linear function classes, the suboptimality of
P3O becomes (cid:101) O
( (cid:112) H 3 d/n ) , where d is the dimension of the feature mapping.
To our best knowledge, we establish the first provably efficient offline RL algorithm for POMDP with a confounded dataset.
Offline
Partial Obs.
Confounded Data
Policy Opt.
Xie et al.
✓ ✘ ✘ ✓
Uehara and Sun (2021) ✓ ✘ ✘ ✓
Jin et al.
(2020) ✘ ✓ ✘ ✓
Efroni et al.
(2022) ✘ ✓ ✘ ✓
Liu et al.
(2022) ✘ ✓ ✘ ✓
Bennett and Kallus (2021) ✓ ✓ ✓ ✘
Shi et al.
✓ ✓ ✓ ✘
P3O (ours) ✓ ✓ ✓ ✓
1. O VERVIEW OF T ECHNIQUES
To deal with the coupled challenges of confounding bias, distributional shift, and large observational spaces, our algorithm and analysis rely on the following technical ingredients.
Confidence regions based on minimax estimation via proximal causal inference.
To handle the confounded offline dataset, we use the proxy variables from proximal causal inference (Lipsitch et al., 2010; Tchetgen et al., 2020; Miao et al., 2018a;b), which allows us to identify the value of each by a set of confounding bridge functions.
These bridge functions only depend on observed variables and satisfy a set of backward conditional moment equations.
We then estimate these bridge functions via minimax estimation (Dikkala et al., 2020; Chernozhukov et al., 2020; Uehara et al., 2021).
More importantly, to handle the distributional shift, we propose a sequence of novel confidence regions for the bridge functions, which quantifies the uncertainty of minimax estimation based on finite data.
This sequence of new confidence regions has not been considered in the previous works on off-policy evaluation (OPE) in POMDPs (Bennett and Kallus, 2021;
Shi et al., 2021) as pessimism seems unnecessary in these works.
Meanwhile, the confidence regions are constructed as a level set w.r.t.
the loss functions of the minimax estimation for bridge functions.
Such a construction contrasts with previous works on offline RL with confidence regions via maximum likelihood estimation (Uehara and Sun, 2021;
Liu et al., 2022) or least square regression (Xie et al., 2021).
Furthermore, we develop a novel theoretical analysis to show that any function in the confidence regions enjoys a fast statistical rate of convergence.
Finally, leveraging the backwardly inductive nature of the bridge functions, our proposed confidence regions and analysis take the temporal structure into consideration, which might be of independent interest to the study on dynamic causal inference (Friston et al., 2003).
Pessimism principle for learning POMDPs.
To learn the optimal policy in the face of distributional shift, we adopt the pessimism principle which is shown to be effective in offline RL in MDPs (Liu et al., 2020; Jin et al., 2021;
Rashidinejad et al., 2021; Uehara and Sun, 2021;
Xie et al., 2021;
Yin and
Wang, 2021; Zanette et al., 2021; Yin et al., 2022; Yan et al., 2022).
Specifically, the newly proposed confidence regions, combined with the identification result based on proximal causal inference, allows us to construct a novel pessimistic estimator for the value of each target policy.
From a theoretical perspective, the identification result and the backward induction property of the bridge functions provide a way of decomposing the suboptimality of the learned policy in terms of statistical errors of the bridge functions.
When combined with the pessimism and the fast statistical rates enjoyed by any functions in the confidence regions, we show that our proposed P3O algorithm efficiently learns the optimal policy under only a partial coverage assumption of the confounded dataset.
We highlight that our work firstly extend the pessimism principle to offline RL in POMDPs with confounded data.
1. R ELATED W ORK
Our work is closely related to the bodies of literature on (i) reinforcement learning POMDPs, (ii) offline reinforcement learning (in MDPs), and (iii) OPE via causal inference.
Compared to the literature, our work simultaneously involve partial observability, confounded data, and offline policy optimization simultaneously, and thus involves the challenges faced by (i)–(iii).
We summarize and contrast with most related existing works in Table 1.
We defer the detailed discussion to Appendix B.
P RELIMINARIES
Notations.
In the sequel, we use lower case letters (i.e., s , a , o , and τ ) to represent dummy variables and upper case letters (i.e., S , A , O , and Γ ) to represent random variables.
We use the variables in the calligraphic font (i.e., S , A , and O ) to represent the spaces of variables, and the blackboard bold font (i.e., P and O ) to represent probability kernels.
2. E PISODIC P ARTIALLY O BSERVABLE M ARKOV D ECISION P ROCESS
We consider an episodic, finite-horizon POMDP, specified by a tuple ( S , O , A , H, µ 1 , P , O , R ) .
Here we let S , A , and O denote the state, action, and observation spaces, respectively.
The integer H ∈ N denotes the length of each episode.
The distribution µ 1 ∈ ∆( S ) denotes the distribution of the initial state.
The set P = { P h } h ∈ [ H ] denotes the collection of state transition kernels where each kernel P h ( ·| s, a ) :
S × A (cid:55)→ ∆( S ) characterizes the distribution of the next state
s h
+1 given that the agent takes action a h = a ∈ A at state s h = s ∈ S and step
h ∈
[ H ] .
The set O =
{ O h } Hh =1 denotes the observation emission kernels where each kernel O h ( ·| s ) : S (cid:55)→ ∆( O ) characterizes the distribution over observations given the current state s ∈ S at step
h ∈
[ H ] .
Finally, the set R = { R h } Hh = denotes the collection of reward functions where each function R h ( s, a ) : S × A (cid:55)→ [0 , 1] specifies the reward the agent receives when taking action a ∈ A at state s ∈ S and step
h ∈
[ H ] .
Different from an MDP, in a POMDP, only the observation o , the action a , and the reward r are observable, while the state variable s is unobservable.
In each episode, the environment first samples an initial state S 1 from µ 1 ( · ) .
At each step h ∈
[ H ] , the environment emits an observation O h from O h ( ·| S h ) .
If an action A h is taken, then the environment samples the next state S h
+1 from P h ( ·| S h , A h ) and assign a reward R h given by R h ( S h , A h ) .
In our setting, we also let O
∈
O denote the prior observation before step h
= 1 .
We assume that O 0 is independent of other random variables in this episode given the first state S 1 .
2. O FFLINE D ATA G ENERATION : C ONFOUNDED D ATASET
Now we describe the data generation process.
Motivated by real-world examples such as precision medicine and autonomous driving discussed in Section 1, we assume that the offline data is generated by some behavior policy π b which has access to the latent states.
Specifically, we let π b = { π bh } Hh
=
denote a collection of policies such that π bh ( ·| s )
: S (cid:55)→ ∆( A ) specifies the probability of taking action each a ∈ A at state s and step h .
This behavior policy induces a set of probability distribution
P b = {P bh } Hh =1 on the trajectories of the POMDP, where P bh is the distribution of the variables at step h when following the policy π b .
Formally, we assume that the offline data is denoted by D
= { ( o k 0 , ( o k 1 , a k 1 , r k 1 ) , · · · , ( o kH , a kH , r kH )) } nk =1 , where n is the number of trajectories, and for each k ∈ [ n ] , ( o k 0 , ( o k 1 , a k 1 , r k 1 ) , · · · , ( o kH , a kH , r kH )) is independently sampled from P b .
We highlight that such an offline dataset is confounded since the latent state S h , which is not stored in the dataset, simultaneously affects the control variables (i.e., action A h ) and the outcome variables (i.e., observation O h and reward R h ).
Such a setting is prohibitive for existing offline RL algorithms for
MDPs as directly applying them will nevertheless incur a confounding bias that is not negligible.
2. L EARNING O BJECTIVE
The goal of offline RL is to learn an optimal policy from the offline dataset which maximizes the expected cumulative reward.
For POMDPs, the learned policy can only depend on the observable information mentioned in Section 2.1.
To formally define the set policies of interest, we first define the space of observable history as H =
{H h } H − 1 h =0 , where each element τ
h ∈
H h is a (partial) trajectory such that τ h ⊆ { ( o 1 , a 1 ) , · · · , ( o h , a h ) } .
We use Γ h to denote the corresponding random variable.
We denote by Π( H ) the class of policies which make decisions based on the current observation o h ∈ O and the history information τ h − 1 ∈ H h − 1 .
That means, a policy π = { π h } H h
=1 ∈ Π( H ) satisfies π h ( ·| o, τ ) :
O×H h − 1 (cid:55)→ ∆( A ) .
The choice of H induces the policy set Π( H ) by specifying the input of the policies.
We now introduce three examples of H and the corresponding Π( H ) .
Example 2.1 (Reactive policy (Azizzadenesheli et al., 2018)) .
The policy only depends on the current observation O h .
Formally, we have H h − 1 = { ∅ } and therefore τ h − 1 = ∅ for each h ∈
[ H ] .
Example 2.2 (Finite-history policy (Efroni et al., 2022)) .
The policy depends on the current observation and the history of length at most k .
Formally, we have H h − 1 =
( O × A ) ⊗ min { k,h − 1 } and τ h − 1 =
(( o l , a l ) , · · · , ( o h − 1 , a h − 1 )) for some k ∈ N , where the index l = max { 1 , h − k } .
Example 2.3 (Full-history policy (Liu et al., 2022)) .
The policy depends on the current observation and the full history.
Formally, H h − 1 =
( O × A ) ⊗ ( h − 1) , and τ h − 1 =
(( o 1 , a 1 ) , · · · , ( o h − 1 , a h − 1 )) .
We illustrate these examples with causal graphs and a more detailed discussion in Appendix C.1.
Now given a policy π ∈ Π( H ) , we denote by J ( π ) the value of π that characterizes the expected cumulative rewards the agent receives by following π .
Formally, J ( π ) is defined as
[Equation]
where γ ∈ (0 , 1] is the discount factor, E π denotes the expectation w.r.t.
P π
= {P πh } Hh =1 which is the distribution of the trajectories induced by π .
We define the suboptimality gap of any policy (cid:98) π as SubOpt
[Equation]
where π ⋆ ∈ arg max π ∈ Π( H
[Equation]
J ( π ) . Here π ⋆ is the optimal policy within Π( H ) .
Our goal is to find some policy (cid:98) π ∈ Π( H ) that minimizes the suboptimality gap in (2.2) based on the offline dataset D .
A LGORITHM : P ROXY V ARIABLE P ESSIMISTIC P OLICY O PTIMIZATION
As is previously discussed, the offline RL problem introduced in Section 2 for POMDPs suffers from three coupled challenges — (i) the confounding bias, (ii) distributional shift, and (iii) large observation spaces.
In the sequel, we introduce an algorithm that addresses all three challenges simultaneously. We first introduce the high-level idea for combating these challenges.
Offline RL for POMDPs is known to be intractable in the worst case (Krishnamurthy et al., 2016).
So we first identify a benign class of POMDPs where the causal structure involving latent states can be captured by only the observable variables available in the dataset D .
For such a class of POMDPs, by leveraging tools from proximal causal inference (Lipsitch et al., 2010; Tchetgen et al., 2020), we then seek to identify the value J ( π ) of the policy π ∈ Π( H ) via some confounding bridge functions
b π (Assumption 3.2) which only depend on the observable variables and thus can be estimated using D (Theorem 3.3).
Identification via proximal causal inference will be discussed in Section 3.1.
In addition, to estimate these confounding bridge functions, we utilize the the fact that these functions satisfy a sequence of conditional moment equations which resembles the Bellman equations in classical MDPs (Bellman and Kalaba, 1965).
Then we adopt the idea of minimax estimation (Dikkala et al., 2020;
Kallus et al., 2021; Uehara et al., 2021; Duan et al., 2021) which formulates the bridge functions as the solution to a series of minimax optimization problems in (3.11).
Additionally, the loss function in minimax estimation readily incorporates function approximators and thus addresses the challenge of large observation spaces.
To further handle the distributional shift, we extend the pessimism principle (Liu et al., 2020; Jin et al., 2021;
Rashidinejad et al., 2021; Uehara and Sun, 2021;
Xie et al., 2021;
Yin and Wang, 2021;
Zanette et al., 2021) to POMDPs with the help of the confounding bridge functions.
In specific, based on the confounded dataset, we first construct a novel confidence region CR π ( ξ ) for b π based on level sets with respect to the loss functions of the minimax estimation (See (3.12) for details).
Our algorithm, Proxy variable Pessimistic Policy Optimization ( P3O ), outputs the policy that maximizes pessimistic estimates the values of the policies within Π( H ) .
The details of P3O is summarized by
Algorithm 1 in Section 3.3.
3. P OLICY V ALUE I DENTIFICATION VIA P ROXIMAL C AUSAL I NFERENCE
To handle the confounded dataset D , we first identify the policy value J ( π ) for each π ∈ Π( H ) using the idea of proxy variables.
Following the notions of proximal causal inference (Tennenholtz et al., 2020; Lipsitch et al., 2010), we assume that there exists negative control actions { Z h } Hh =1 and negative control outcomes { W h } Hh =1 satisfying the following independence assumption.
Assumption 3.1 (Negative control) .
We assume there exist negative control variables { W h } Hh =1 and { Z h } Hh =1 measurable with respect to the observed trajectories, such that under P b , it holds that
[Equation]
We explain in detail the existence of such negative control variables for all the three different choices of history H in Examples 2.1, 2.2, and 2.3 respectively in Appendix C.1.
Besides Assumption 3.1, our identification of policy value also relies on the notion of confounding bridge functions (Kallus et al., 2021; Shi et al., 2021), for which we make the following assumption.
Assumption 3.2 (Confounding bridge functions) .
For any history-dependent policy π ∈ Π( H ) , we assume the existence of the value bridge functions { b πh : A × W
(cid:55)→ R } Hh =1 and the weight bridge functions { q πh : A × Z (cid:55)→ R } Hh =1 which are defined as the solution to the following equations almost surely with respect to the measure P b :
[Equation]
Here b πH +1 is a zero function and µ h ( S h , Γ h − 1 ) in (3.3) is defined as the importance sampling ratio µ h
( S h , Γ h − 1 ) :
= P πh ( S h , Γ h − 1 ) / P bh ( S h , Γ h − 1 ) .
We use “confounding bridge function” and “bridge function” interchangeably throughout the paper.
We remark that in the proximal causal inference literature, the existence of such bridge functions bears more generality than assuming certain complicated completeness conditions (Cui et al., 2020), as discussed by Kallus et al.
The existence of such bridge functions is justified by conditions on the the rank of certain conditional probabilities or singular values of certain conditional expectation linear operators.
See Appendix C.2 for examples of Assmuption 3.2 under previous choice of H .
Now given Assumption 3.1 and 3.2 on the existence of proxy variables and bridge functions, we are ready to present the main identification result.
It represents the true policy value J ( π ) via the value bridge functions, as is concluded in the following theorem.
Theorem 3.3 (Identification of policy value) .
Under Assumption 3.1 and 3.2, for any historydependent policy π ∈ Π( H ) , it holds that
[Equation]
See Appendix E for a detailed proof.
Note that although we have assumed the existence of both the value bridge functions in (3.2) and the weight bridge functions in (3.3), Theorem 3.3 represents J ( π ) using only the value bridge functions.
In (3.2) all the random variables involved are observed by the learner and distributed according to the data distribution P b , which means that the value bridge functions can be estimated from data.
This overcomes the confounding issue.
3. P OLICY E VALUATION VIA M INIMAX E STIMATION WITH U NCERTAINTY Q UANTIFICATION
According to Theorem 3.3 and Assumption 3.2, to estimate the value J ( π ) of π ∈ Π( H ) , it suffices to estimate the value bridge functions { b πh } Hh =1 by solving (3.2), which is a conditional moment equation.
To this end, we adopt the method of minimax estimation (Dikkala et al., 2020; Uehara et al., 2021; Duan et al., 2021).
Furthermore, in order to handle the distributional shift between behavior policy and target policies, we construct a sequence of confidence regions for { b πh } Hh =1 based on minimax estimation, which allows us to apply the pessimism principle by finding the most pessimistic estimates within the confidence regions.
Specifically, minimax estimation involves two function classes B ⊆ { b : A × W (cid:55)→ R } and G ⊆
{ g : A × Z (cid:55)→ R } , interpreted as the primal and dual function classes, respectively.
Theoretical assumptions on B and G are presented in Section 4.
In order to find functions that satisfy (3.2), it suffices to find b =
( b 1 , · · · , b H ) ∈ B ⊗ H such that the following conditional moment
[Equation]
is equal to zero almost surely for all h ∈
[ H ] , where b H +1 is a zero function.
Intuitively, the quantity (3.5) can be interpreted as the “Bellman residual" of the value bridge functions b .
Notice that (3.5) being zero almost surely for all h ∈
[ H ] is actually a conditional moment equation, which is equivalent to finding b ∈ B ⊗ H such that the following residual mean squared error (RMSE) is minimized for all h :
[Equation]
It might seem tempting to directly minimize the empirical version of (3.6).
However, this is not viable as one would obtain a biased estimator due to an additional variance term.
The reason is that the quantity defined by (3.5) is a conditional expectation and therefore RMSE defined by (3.6) cannot be directly unbiasedly estimated from data (Farahmand et al., 2016).
In the sequel, we adopt the technique of minimax estimation to circumvent this issue.
In particular, we first use Fenchel duality to write (3.6) as
[Equation]
which holds when the dual function class G is expressive enough such that ℓ πh ( b h , b h +1 ) / 2 λ ∈ G .
Then thanks to the interchangeability principle (Rockafellar and Wets, 2009; Dai et al., 2017; Shapiro et al., 2021), we can interchange the order of maximization and expectation and derive that
[Equation]
The core idea of minimax estimation is to minimize the empirical version of (
[Equation]
8) instead of (3.6), and the benefit of doing so is a fast statistical rate of (cid:101) O
( n − 1 / 2 ) (Dikkala et al., 2020; Uehara et al., 2021), as we can see in the sequel.
For simplicity, in the following, we define Φ λπ,h : B × B × G (cid:55)→ R with parameter λ > 0 as
[Equation]
and we denote by (cid:98) Φ λπ,h :
B × B × G (cid:55)→ R the empirical version of Φ λπ,h , i.e.,
[Equation]
where (cid:98) E π b denotes the empirical version of E π b based on dataset D described in Section 2.2.
Furthermore, note that the value bridge functions ( b π 1 , · · · , b πh ) admit a sequential dependence structure.
To handle such dependency, for any π ∈ Π( H ) , h ∈ [ H ] , and b h +1 ∈ B , we first define the minimax estimator (cid:98) b h ( b h +1 ) as
[Equation]
[Equation]
Based on (3.11), we propose a confidence region for b π :=
( b π 1 , · · · , b πH ) ∈ B ⊗ H as
[Equation]
[Equation]
From the above definition 1 , one can see that CR π ( ξ ) is actually a coupled sequence of H confidence regions, where each single confidence region aims to cover a function b πh .
For notational simplicity, we use a single notation CR π ( ξ ) to denote all the H confidence regions.
Intuitively, the confidence region CR π ( ξ ) contains all b ∈ B ⊗ H whose RMSE does not exceed that of ( (cid:98) b h ( b h +1 ) , b h +1 ) by too much at each h ∈
[ H ] .
The confidence region takes the sequential dependence of confounding bridge functions into consideration in the sense that each b ∈ CR π ( ξ ) is restricted through the minimax estimation loss between continuous steps.
As we show in Section D, with high probability, the confidence region CR π ( ξ ) contains the true bridge value functions b π .
More importantly, every b ∈ CR π ( ξ ) enjoys a fast statistical rate of (cid:101)
O
( n − 1 / 2 ) .
Now combining the confidence region (3.12) and the identification formula (3.4), for any policy π ∈ Π( H ) , we adopt an pessimistic estimate of the value of J ( π ) as (cid:98) J Pess ( π ) :
=
min b ∈ CR π ( ξ ) (cid:98) F ( b ) , where (cid:98) F ( b ) := (cid:98) E π b (cid:34)(cid:88) a ∈A
[Equation] [Equation]
3. P OLICY O PTIMIZATION
Given the pessimistic value estimate (3.13), P3O chooses (cid:98) π which maximizes (cid:98) J Pess ( π ) , that is,
[Equation]
π
[Equation]
We summarize the entire P3O algorithm in Algorithm 1.
In Section 4, we show that under some mild assumptions on the function classes B and G and under only a partial coverage assumption on the dataset D , the suboptimality (2.2) of Algorithm 1 decays at the fast statistical rate of (cid:101) O
( n − 1 / 2 ) , where (cid:101) O ( · ) omits H and factors that characterize the complexity of the function classes.
T HEORETICAL R ESULTS
In this section, we present our theoretical results.
For ease of presentation, we first assume that both the primal function class B and the policy class Π( H ) are finite sets with cardinality | B | and | Π( H ) | , respectively.
But we allow the dual function class G to be an infinite set.
Our results can be easily extended to infinite B and Π( H ) using the notion of covering numbers (Wainwright, 2019), which we demonstrate with linear function approximation in Section H.1.
We first introduce some necessary assumptions for efficient learning of the optimal policy.
To begin with, the following Assumption 4.1 ensures that the offline data generated by π b has a good coverage over π ⋆ .
The problem would become intractable without such an assumption (Chen and Jiang, 2019).
Assumption 4.1 (Partial coverage) .
We assume that the concentrability coefficient for the optimal policy π ⋆ , defined as C π ⋆ := max h ∈
[ H ] E π b (cid:2) ( q π ⋆ h
( A h , Z h )) 2 (cid:3) , satisfies that C π ⋆ < + ∞ .
Very importantly, Assumption 4.1 only assumes the partial coverage, i.e., the optimal policy π ⋆ is well covered by π b (Jin et al., 2021; Uehara and Sun, 2021), which is significantly weaker than the uniform coverage, i.e., the entire policy class Π( H ) is covered by π b (Chen and Jiang, 2019) in the sense that max π ∈ Π( H )
C π < + ∞ .
See Appendix B for more about partial coverage in POMDP.
The next assumption is on the functions classes B and G .
We require that B and G are uniformly bounded, and that G is symmetric, star-shaped, and has bounded localized Rademacher complexity.
Assumption 4.2 (Function classes B and G ) .
We assume the classes B and G satisfy that: i)
There exist M B , M G < + ∞ such that B , G are bounded by sup b ∈ B sup w ∈W | (cid:80) a ∈A b ( a, w ) | ≤ M B 2 , sup g ∈ G sup ( a,z ) ∈A×Z | g ( a, z ) | ≤ M G ; ii) G is star-shaped, i.e., for any g ∈ G and λ ∈ [0 , 1] , it holds that λg ∈ G ; iii) G is symmetric, i.e., for any g ∈ G , it holds that − g ∈ G ; iv)
For any step h ∈
[ H ] , G has bounded critical radius α G ,h,n which solves inequality R n ( G ; α ) ≤ α 2 /M G , where R n ( G , α ) is the localized population
Rademacher complexity of G under the distribution of ( A h , Z h ) induced by π b , that is,
[Equation]
with ∥ g ∥ 2 defined as ( E π b [ g 2 ( A h , Z h )]) 1 / 2 , random variables { ϵ i } ni =1 independent of ( A h , Z h ) and independently uniformly distributed on { +1 , − 1 } .
Also, we denote α G ,n := max h ∈
[ H ] α G ,h,n .
Finally, to ensure that the minimax estimation in Section 3.2 learns the value bridge functions unbiasedly, we make the following completeness and realizability assumptions on the function classes B and G which are standard in the literature (Dikkala et al., 2020; Xie et al., 2021; Shi et al., 2021).
Assumption 4.3 (Completeness and realizability) .
We assume that, i) completeness: for any h ∈
[ H ] , any π ∈ Π( H ) , and any b h , b h +1 ∈ B , it holds that 12 λ ℓ πh ( b h , b h +1 ) ∈ G where ℓ πh is defined in (3.5); ii) realizability: for any h ∈
[ H ] , any π ∈ Π( H ) , and any b h +1 ∈ B , there exists b ⋆ ∈ B such that
L πh ( b ⋆ , b h +1 ) ≤ ϵ B for some ϵ B < + ∞ , i.e., we assume that
[Equation]
Here the completeness assumption means the dual function class G is rich enough which guarantees the equivalence between L πh ( · , · ) and max g ∈ G
Φ λπ,h ( · , · ; g ) .
The realizability assumption means that the primal function class B is rich enough such that (3.2) always admits an (approximate) solution.
With these technical assumptions, we can establish our main theoretical results in the following theorem, which gives an upper bound of the suboptimality (2.2) of the policy (cid:98)
π output by Algorithm 1.
Theorem 4.4 (Suboptimality) .
Under Assumptions 3.1, 3.2, 4.1, 4.2, and 4.3, by setting the regularization parameter λ and the confidence parameter ξ as λ = 1 and
[Equation]
then probability at least 1 − 3 δ , it holds that
[Equation]
where ζ = min { δ, 4 c 1 exp( − c 2 nα 2 G ,n ) } .
Here C π ⋆ , α G ,n , and ϵ B are defined in Assumption 4.1, 4.2, and 4.3, respectively.
And C 1 , C ′ 1 , c 1 , and c 2 are some problem-independent universal constants.
We introduce all the key technical lemmas and sketch the proof of Theorem 4.4 in Section D.
We refer to Appendix G for a detailed proof.
When it holds that α G ,n = (cid:101) O ( n − 1 / 2 ) and ϵ B = 0 , Theorem 4.4 implies that SubOpt ( (cid:98) π ) ≤ (cid:101) O
( n − 1 / 2 ) , which corresponds to a “fast statistical rate” for minimax estimation (Uehara et al., 2021).
The derivation of such a fast rate relies on a novel analysis for the risk of functions in the confidence region, which is shown by Lemma D.3 in Section D.
Meanwhile, for many choices of the dual function class G , it holds that α G ,n scales with √ log N G where N G denotes the complexity measure of the class G .
In such cases, the suboptimality also scales with √ log N G , without explicit dependence on the cardinality of the spaces S , A , or O .
Finally, we highlight that, thanks to the principle of pessimism, the suboptimality of P3O depends only on the partial coverage concentrability coefficient C π ⋆ , which can be significantly smaller than the uniform coverage concentrability
coefficient sup π ∈ Π( H ) C π .
In conclusion, when α G ,n = (cid:101) O ( (cid:112) log N G /n ) and ϵ B = 0 , the P3O algorithm enjoys a (cid:101)
O ( H (cid:112) C π
⋆ log N G /n ) suboptimality.
Linear function approximation.
Theorem 4.4 can be readily extended to the case of linear function approximation (LFA) with infinite-cardinality B and Π( H ) , which yields an (cid:101) O
( (cid:112) H 3 d/n ) suboptimality.
Due to space limit, we defer the detailed setup and main results of LFA to Appendix H.1.