Provably Efficient Reinforcement Learning in Partially Observable Dynamical Systems
Abstract
We study Reinforcement Learning for partially observable dynamical systems using function approximation.
We propose a new Partially Observable Bilinear
Actor-Critic framework , that is general enough to include models such as observable tabular Partially Observable Markov Decision Processes (POMDPs), observable Linear-Quadratic-Gaussian (LQG), Predictive State Representations (PSRs), as well as a newly introduced model Hilbert Space Embeddings of
POMDPs and observable POMDPs with latent low-rank transition.
Under this framework, we propose an actor-critic style algorithm that is capable of performing agnostic policy learning.
Given a policy class that consists of memory based policies (that look at a fixed-length window of recent observations), and a value function class that consists of functions taking both memory and future observations as inputs, our algorithm learns to compete against the best memory-based policy in the given policy class.
For certain examples such as undercomplete observable tabular POMDPs, observable LQGs and observable POMDPs with latent low-rank transition, by implicitly leveraging their special properties, our algorithm is even capable of competing against the globally optimal policy without paying an exponential dependence on the horizon in its sample complexity.
Introduction
Large state space and partial observability are two key challenges of Reinforcement Learning (RL).
While recent advances in RL for fully observable systems have focused on the challenge of scaling RL to large state space in both theory and in practice using rich function approximation, the understanding of large-scale RL under partial observability is still limited.
In POMDPs, for example, a core issue is that the optimal policy is not necessarily Markovian since the observations are not Markovian
A common heuristic to tackle large-scale RL with partial observability in practice is to simply maintain a time window of the history of observations, which is treated as a state to feed into the policy and the value function.
Such a window of history can be often maintained explicitly via truncating away older history (e.g., DQN uses a window with length 4 for playing video games [52]; Open AI Five uses a window with length 16 for LSTMs [5]).
Since even for planning under partial observations and known dynamics, finding the globally optimal policy conditional on the entire history is generally NP-hard (due to the curse of the history)
[46, 55, 23], searching for a short memory-based policy can be understood as a reasonable middle ground that balances computation and optimality.
The impressive empirical results of these prior works also demonstrate that in practice, there often exists a high-quality policy (not necessarily the globally optimal) that is only a function of a short window of recent observations.
However, these prior works that search for the best memorybased policy unfortunately cannot ensure sample efficient PAC guarantees due to the difficulty of strategic exploration in POMDPs.
The key question that we aim to answer in this work is:
Can we design provably efficient RL algorithms that agnostically learn the best fixed-length memory-based policy with function approximation?
We provide affirmative answers to the above question.
More formally, we study RL for partially observable dynamical systems that include not only the classic Partially Observable MDPs (POMDPs) [53, 56, 58], but also a more general model called Predictive State Representations (PSRs) [45].
We design a model-free actor-critic framework, named PO-Bilinear Actor-Critic Class , where we have a policy class (i.e., actors) that consists of policies that take a fixed-length window of observations as input (memory-based policy), and a newly introduced future-dependent value function class (i.e., critics) that consists of functions that take the fixed-length window of history and (possibly multi-step if the system is overcomplete) future observations as inputs.
A future-dependent value function class is an analog of the value function class tailored to partially observable systems that only relies on observable quantities (i.e., past and future observations and actions).
In our algorithm, we agnostically search for the best memory-based policy from the given policy class.
Our framework is based on the idea of a newly introduced notion of future-dependent value function equipped with future observations.
While the idea of using future observations has been used in the literature on POMDPs, our work is the first to use this idea to learn a high-quality policy in a model-free manner.
Existing works discuss how to use future observations only in a model-based manner [8, 27].
By leveraging these model-based viewpoints, while recent works discuss strategic exploration to learn near-optimal policies, their results are either limited to the tabular setting (and are not scalable for large state spaces)
[36, 24, 3, 74, 47] or are tailored to specific non-tabular models and unclear how to incorporate general function approximation [60, 42, 9].
We break these barriers by devising a new actor-critic-based model-free view on POMDPs.
We demonstrate the scalability and generality of our PO-bilinear actor-critic framework by showing PAC-guarantee on many models as follows (see Table 1 for a summary).
Observable Tabular POMDPs.
In tabular observable POMDPs, i.e., POMDPs where multistep future observations retain information about latent states, the PO-bilinear rank decomposition holds.
We can ensure the sample complexity is Poly( S, A M , O M , A K , O K , H, 1 /σ 1 ) where σ 1 = min
x
∥
O x ∥ 1 / ∥ x ∥ 1 ( O is an emission matrix),and S, A,
O are the cardinality of state, action, observation space, respectively, H is the horizon, and K is the number of future observations.
In the special undercomplete ( O ≥ S ) case, our framework is also flexible enough to set the memory length according to the property of the problems in order to search for the globally optimal policy.
More specifically, using the latest result from [23] about belief contraction, we can set
M = ˜
O ((1 /σ 41 ) ln( SH/ϵ ))
with ϵ being the optimality threshold.
This allows us to compete against the globally optimal policy without paying an exponential dependence on H .
Observable Linear Quadratic Gaussian (LQG).
In observable LQG – a classic partial observable linear dynamical system, our algorithm can compete against the globally optimal policy with a sample complexity scaling polynomially with respect to the horizon, dimensions of the state, observation, and action spaces (and other system parameters).
This is achieved by simply setting the memory length M to H .
The special linear structures of the problem allow us to avoid exponential dependence on H even when using the full history as a memory.
While the global optimality results in tabular POMDPs and LQG exist by using different algorithms, to the best of our knowledge, this is the first unified algorithm that can solve both tabular POMDPs and LQG simultaneously without paying an exponential dependence on horizon H .
Observable Hilbert Space Embedding POMDPs (HSE-POMDPs).
Our framework ensures the agnostic PAC guarantee on HSE-POMDPs where policy induced transitions and omission distributions have condition mean embeddings [8, 7].
This model naturally generalizes tabular POMDPs and
LQG.
We show that the sample complexity scales polynomially with respect to the dimensions of the embeddings.
This is the first PAC guarantee in HSE-POMDPs.
Predictive State Representations (PSRs).
We give the first PAC-guarantee on PSRs.
PSRs model partially observable dynamical systems without even using the concept of latent states and strictly generalize the POMDP model.
Our work significantly generalizes a prior PAC learning result for
[Table]
reactive PSRs (i e., reactive PSRs require a strong condition that the optimal policy only depends on the latest observation) which is a much more restricted setting [33].
M -step decodable POMDPs [19].
Our framework can capture M -step decodable POMDPs where there is a (unknown) decoder that can perfectly decode the latent state by looking at the latest Mmemory.
Our algorithm can compete against the globally optimal policy with the sample complexity scaling polynomially with respect to horizon H , S, A M , and the statistical complexities of function classes, without any explicit dependence on O .
This PAC result is similar to the one from [19].
Observable POMDPs with low-rank latent transition.
Our framework captures observable
POMDPs where the latent transition is low-rank.
This is the first PAC guarantee in this model.
Under this model, we first show that with M =
˜
O (cid:0) (1 /σ 41 )
ln( dH/ϵ ) (cid:1) where d is the rank of the latent transition matrix, there exists an M -memory policy that is ϵ -near optimal with respect to the globally optimal policy.
Then, starting with a general model class that contains the ground truth transition and omission distribution (i.e., realizability in model class), we first convert the model class to a policy class and a future-dependent value function class, and we then show that our algorithm competes against the globally optimal policy with a sample complexity scaling polynomially with respect to H, d, |A| (1 /σ 41 )ln( dH/ϵ ) , 1 /σ 1 , and the statistical complexity of the model class.
Particularly, the sample complexity has no explicit dependence on the size of the state and observation space, instead it just depends on the statistical complexity of the given model class.
1.1 Related Works
We discuss related works about online RL for POMDPs.
Additional works related to system identification, generalization and function approximation of RL in MDPs, PSRs and future-dependent value functions are provided in Section A.
Prior works [38, 20] showed A H -type sample complexity bounds for general POMDPs.
Exponential dependence can be circumvented with more structures.
First, in the tabular setting, under observability assumptions, in [3, 24, 34, 47, 22], favorable sample complexities are obtained by leveraging the spectral learning technique [29] (see section 1.1 in [34] for an excellent summary).
Second, in
LQG, which is a partial observable version of LQRs, in [42, 60], sub-linear regret algorithms are proposed.
These works use random policies for exploration, which is sufficient for LQG.
Since random exploration strategy is not enough for tabular POMDPs, it is unclear if the existing techniques from LQG can be applied to solve general POMDPs.
Third, the recent work [19] provides a new model called M -step decodable POMDP (when M = 1 , it is Block MDP) with an efficient algorithm.
Our framework captures all above mentioned POMDP models.
In addition, we propose a new model called HSE-POMDPs which extends prior works on HSE-HMM[7] to POMDPs and includes LQG and tabular POMDPs.
Our algorithm delivers the first PAC bound for this model.
Finally, we remark it is unclear whether our framework can capture several existing POMDP models [9, 41].
Preliminary
We introduce the background for POMDPs.
We consider an episodic POMDP specified by M = ⟨S , O , A , H, T , O ⟩ , where S is the unobserved state space, O is the observation space,
A is the action space, H is the horizon, T : S ×
A → ∆( S ) is the transition probability, O : S → ∆( O ) is the emission probability, and r : O × A → R is the reward.
Here, T , O are unknown distributions.
For notation simplicity, we consider the time-homogeneous case in this paper; Extension to the time-inhomogeneous setting is straightforward.
[Figure]
In our work, we consider M -memory policies.
Let Z h
=
( O × A ) min { h,M } and ¯ Z h
=
Z h − 1 × O .
An element z h ∈
Z h is represented as z h =
[ o max( h − M +1 , 1): h , a max( h − M +1 , 1): h ] , and an element ¯ z h ∈
¯ Z h is represented as ¯ z h =
[ o max( h − M, 1): h , a max( h − M, 1): h − 1 ] (thus, ¯ z h =
[ z h − 1 , o h ] ).
Fig- ure 2 illustrates this situation.
An M -memory policy is defined as π = { π h } Hh =1 where each π h is a mapping from ¯ Z h to a distribution over actions ∆( A ) .
In a POMDP, an M -memory policy generates the data as follows.
Each episode starts with the initial state s 1 sampled from some unknown distribution.
At each step h ∈
[ H ] , from s h ∈ S , the agent observes o h ∼ O ( ·| s h ) , executes action a h ∼ π h ( ·| ¯ z h ) , receives reward r ( s h , a h ) , and transits to the next latent state s h +1 ∼ T ( ·| s h , a h ) .
Note that the agent does not observe the underlying states but only the observations { o h } h ≤ H .
We denote J ( π ) as the value of the policy π , i.e., E [ (cid:80)
Hh =1 r h ; a 1: H ∼ π ] where the expectation is taken w.r.t.
the stochasticity of the policy π , emissions distribution O and transition dynamics T .
We define a value function for a policy π at step h to be the expected cumulative reward to go under the policy π starting from a z ∈ Z h − 1 and s ∈ S , i.e. V πh :
Z h − 1 × S → R where V πh ( z, s ) =
E [ (cid:80) Hh ′ =
h r h ′ |
z h − 1 =
z, s h
= s ; a h : H ∼ π ] .
The notation E [ · ; a h : H ∼ π ] means the expectation is taken under a policy π from h to H .
Compared to the standard MDP setting, the expectation is conditional on not only s h but also z h − 1 since we consider M -memory policies.
The corresponding
Bellman equation for V πh is V πh ( z h − 1 , s h ) =
E (cid:2) r h + V πh +
( z h , s h +1 )
| z h − 1 , s h ; a h ∼ π (cid:3)
The Actor-critic function approximation setup.
Our goal is to find a near optimal policy that maximizes the policy value J ( π ) in an online manner.
Since any POMDPs can be converted into
MDPs by setting the state at level h to the observable history up to h , any off-the-shelf online provably efficient algorithms for MDPs can be applied to POMDPs.
By defining H h as the whole history up to step h ∈
[ H ] (i.e., a history τ
h ∈
H h is in the form of o 1: h , a 1: h − 1 ) , these naïve algorithms ensure that output policies can compete against the globally optimal policy π ⋆ gl = argmax π ∈ ¯Π J ( π ) where ˜Π = { ¯Π h } , ˜Π h
= [ H h → ∆( A )] .
However, this conversion results in the error with exponential dependence on the horizon H , which is prohibitively large in the long horizon setting.
Instead of directly competing against the globally optimal policy, we aim for agnostic policy learning , i.e., compete against the best policy in a given M -memory policy class.
Our function approximation setup consists of two function classes, ( a ) A policy class Π consisting of M -memory policies Π :=
{ Π h } Hh =1 where Π h ⊂ [ ¯ Z h → ∆( A )] (i.e., actors), ( b ) A set of value future-dependent value functions G = {G h } Hh =1 where G h ⊂ [ ¯ Z h → R ] , whose role is to approximate V πh (i.e., critics).
Our goal is to provide an algorithm that outputs a policy ˆ π = { ˆ π h } that has a low excess risk, where excess risk is defined by R ( π ) :
=
J (ˆ π ) − J ( π ⋆ ) where π ⋆ = argmax π ∈ Π J ( π ) is the best policy in class Π .
To motivate this agnostic setting, M -memory policies are also widely used in practice, e.g.,
DQN [52] sets M = 4 .
Besides, there are natural examples where M -memory policies are close to the globally optimal policy with M being only polynomial with respect to other problem dependent parameters, e.g., observable POMDPs [23] and LQG [42, 60, 48].
We will show the global optimality in these two examples later, without any exponential dependence on H in the sample complexity.
Remark 1 (Limits of existing MDP actor-critic framework) .
While general actor-critic framework proposed in MDPs [33] is applicable to POMDPs via the naïve POMDP to MDP reduction, it is unable to leverage any benefits from the restricted policy class.
This naïve reduction (from POMDP to MDP) uses full history and will incur sample complexity that scales exponentially in the horizon.
Additional notation.
Let [ H ] = { 1 , · · · , H } and [ t ] = { 1 , · · · , t } .
Give a matrix A , we denote its pseudo inverse by A † and the operator norm by ∥ A ∥ .
We define the ℓ 1 norm ∥
A ∥ 1 = max x : x ̸ =0 ∥
Ax ∥ 1 / ∥ x ∥ 1 .
The outer product is denoted by ⊗ .
Let d πh
( · ) ∈ ¯ Z h × S be the marginal distribution at h and δ ( · ) be the Dirac delta function.
We denote the policy δ ( a = a ′ ) by do( a ′ ) .
We denote a uniform action by U ( A ) .
Given a function class G , we define ∥G∥ ∞ = sup g ∈G ∥ g ∥ ∞ .
Future-Dependent Value Functions and the PO-bilinear Framework
Unlike MDPs, we cannot directly work with value functions V πh ( s ) in POMDPs, since they depend on the unobserved state s .
To handle this issue, below we first introduce new future-dependent value functions by using future observations, and then discuss the PO-bilinear framework.
3.1 Future-Dependent Value Functions
Definition 1 (K-step future-dependent value functions) .
Fix a set of policies π out =
{ π outi } Ki = where π outi : O → ∆( A ) .
Value future-dependent value functions g πh : Z h − 1 ×
O K × A K − 1 → R at step h ∈
[ H ] for a policy π are defined as the solution to the following integral equation: ∀
[Equation]
where the expectation is taken under the policy π out .
Future-dependent value functions do not necessarily exist, nor are needed to be unique.
At an intuitive level, K-step future-dependent value functions are embeddings of the value functions onto the observation space, and its existence essentially means that K-step futures have sufficient information to recover the latent state dependent value function.
The proper choice of π out would depend on the underlying models.
For example, we use uniform policy in the tabular case, and δ ( a = 0) in
LQG.
For notational simplicity, we mostly focus on the case of K = 1 , though we will also discuss the general case of K ≥ 2 .
The simplified definition for 1-step future-dependent value functions is provided in the following.
Note that this definition is agnostic to π out .
Definition 2 (1-step future-dependent value functions) .
One-step future-dependent value functions g πh : Z h − 1 × O
→ R at step h ∈ [ H ] for a policy π are defined as the solution to the following:
[Equation]
In Section 4, we will demonstrate the form of the future-dependent value function for various examples.
The idea of encoding latent state information using the statistics of (multi-step) futures have been widely used in learning models of HMMs [63, 29], PSRs [8, 7, 27, 67], and system identification [72].
Existing provably efficient (online) RL works for POMDPs elaborate on this viewpoint [36, 24, 3].
Compared to them, the novelty of future-dependent value functions is that it is introduced to recover value functions but not models .
This model-free view differs from the existing dominant model-based view in online RL for POMDPs.
In our setup, we can control systems if we can recover value functions on the underlying states even if we fail to identify the underlying model.
3.2 The PO-Bilinear Actor-critic Framework for POMDPs
With the definition of future-dependent value functions, we are now ready to introduce the PO-bilinear actor-critic (AC) class for POMDPs.
We will focus on the case of K = 1 here.
Let G =
{G h } Hh =1 , where G h ⊂ [ ¯ Z h → R ] , be a class consisting of functions that satisfy the following realizability assumption w.r.t.
the policy class Π .
Assumption 1 (Realizability) .
We assume that G is realizable w.r.t.
the policy class Π , i.e., ∀ π ∈ Π , h ∈
[ H ] , there exists at least one g π h ∈ G h such that g π h is a future-dependent value function w.r.t.
the policy π .
Note that realizability implicitly requires the existence of ( g π h ) .
We next introduce the PO-Bilinear Actor-critic class.
For each level h ∈ [ H ] , we first define the
Bellman loss:
[Equation]
(¯ z h +1 ) : a 1: h − 1 ∼ π in , a h ∼ π ] given M-memory policies π = { π h } , π in = { π inh } and g = { g h } .
Letting g π = { g πh } Hh =1 be a future-dependent value function for π , our key observation is that future-dependent value functions satisfy 0 =
Br h ( π, g π ; π in ) for any M memory roll-in policy π in = { π inh } Hh =1 , and any evaluation pair ( π, g π ) .
This is an analog of Bellman equations on MDPs.
The above equation tells us that
Br h ( π, g ; π in ) is a right loss to quantify how much the estimator g is different from g πh .
When
Br h ( π, g ; π in ) has a low-rank structure in a proper way, we can efficiently learn a near-optimal M memory policy.
The following definition precisely quantifies the low-rank structure that we need for sample efficient learning.
Definition 3 (PO-bilinear AC Class, K = 1 ) .
The model is a PO-bilinear Actor-critic class of rank d if G is realizable, and there exist W h : Π × G → R d and X h : Π → R d such that for all π ′ , π ∈ Π , g ∈ G and h ∈
[ H ] ,
E [ g h (¯ z h ) − r h − g h +
(¯ z h +1 )
; a 1: h − 1 ∼ π ′ , a h ∼ π ] = ⟨ W h ( π, g ) , X h ( π ′ ) ⟩ . 2.
W h
( π, g π ) =
for any π ∈ Π and the corresponding future-dependent value function g π ∈ G .
We define d as the PO-bilinear rank.
Remark 2 (Two Important Extensions) .
While the above definition is enough to capture most of the examples we discuss later in this work, including undercomplete tabular POMDPs, LQG, HSEPOMDPs, we provide two useful extensions.
The first extension incorporates discriminators into the framework, which can be used to capture the M-step decodable POMDPs and POMDPs with low-rank latent transition (see Section F).
The second extension incorporates multi-step futures, which can be used to capture overcomplete POMDPs and general PSRs (see Section B.
Examples of PO-Bilinear Actor-critic Classes
We consider three examples (observable tabular POMDPs, LQG, HSE-POMDPs) that admit PObilinear rank decomposition.
Our framework can also capture PSRs, M -step decodable POMDPs and low rank observable POMDPs, of which the discussions are deferred to Section E, G.1 and G.2, respectively.
We mainly focus on one-step future, i.e., K = 1 , and briefly discuss the extension to
K > 1 in the tabular case.
In this section, except for LQG, we assume r h ∈ [0 , 1] for any h ∈
[ H ] .
All the missing proofs are deferred to Section C.
4.1 Observable Undercomplete Tabular POMDPs
Example 1 (Observable undercomplete tabular POMDPs) .
Let O ∈ R |O|×|S| where the entry indexed by a pair ( o, s ) is defined as O o,s
= O
( o | s ) .
Assume that rank( O ) = |S| , which we call observability.
This requires undercompletenes |O| ≥ |S| .
The following lemma shows that O being full rank implies the existence of future-dependent value functions g πh .
Lemma 1.
For Example 1, there exists a g πh satisfying Definition 2 for any π ∈ Π and h ∈
[ H ] .
Proof.
Consider any function f : Z h − 1 × S → R (thus, this captures all possible V πh ).
Denote 1 ( z ) as the one-hot encoding of z over Z h − 1 (similarly for 1 ( s ) ).
We have f ( z, s ) =
⟨ f, 1 ( z ) ⊗ 1 ( s ) ⟩ = ⟨ f, 1 ( z ) ⊗ ( O † O 1 ( s )) ⟩ , where we use the assumption that rank( O ) = |S| and thus O †
O
=
I .
Then,
[Equation]
which means g πh ( z, o ) :
= ⟨ V πh , 1 ( z ) ⊗ O † 1 ( o ) ⟩ .
We next show that the PO-Bilinear rank (Definition 3) is bounded by |S| ( |O||A| ) M .
Lemma 2.
Assume O is full column rank.
Set the future-dependent value function class G h
=
[ Z h − 1 × O → [0 , C G ]] for certain C G ∈ R , and policy class Π h
=
[ ¯ Z h → ∆( A )] .
Then, the model is a PO-biliner AC class (Definition 3) with PO-bilinear rank at most |S| ( |O||A| ) M .
Later, we will see that the PO-bilinear rank in the more general definition is just |S| in Section F.
This fact will result in a significant improvement in terms of the sample complexity, and will result in a sample complexity that does not incur |O| M .
Lastly, we touch on overcomplete POMDPs ( |O| ≤ |S| ) when we use multi-step futures.
For details, refer to Section C.2.
In this case, the existence of future-dependent value function is ensured when |O| K |A| K − 1 × |S| matrix with entries equal to P ( o h : h + K − 1 , a h : h + K − 2 | s h ; a h : h + K − 2 ∼ U ( A )) .
4.2 Observable Linear Quadratic Gaussian
The next example is Linear Quadratic Gaussian (LQG) with continuous state and action spaces.
The details are deferred to Section M. Here, we set M = H − 1 so that the policy class Π contains the globally optimal policy.
Example 2 (Linear Quadratic Gaussian (LQG)) .
Consider LQG:
[Equation]
where ϵ, τ are Gaussian distribution with mean 0 and variances Σ ϵ and Σ τ , respectively, and s ∈ R d s , o ∈ R d o , and a ∈ R d a , and Q, R are positive definite matrices.
We define the policy class as the linear policy class Π h
= { δ ( a h =
K h ¯ z h ) | K
h ∈ R |A|× d ¯ zh ) , where d ¯ z h is a dimension of ¯ z h ∈ ¯ Z h .
This choice is natural since the globally optimal policy is known to be linear with respect to the entire history [6, Chapter 4].
We define two quadratic features, ϕ h ( z h − 1 , s h ) =
(1 , [ z ⊤ h − 1 , s ⊤ h ] ⊗ [ z ⊤ h − 1 , s ⊤ h ])
⊤ with z h − 1 ∈
Z h − 1 , s h ∈ S , and ψ h ( z h − 1 , o h ) =
(1 , [ z ⊤ h − 1 , o ⊤ h ] ⊗ [ z ⊤ h − 1 , o ⊤ h ])
⊤ with z h − 1 ∈
Z h − 1 , o h ∈
O .
We have the following lemma.
Lemma 3 (PO-bilinear rank of observable LQG) .
Assume rank( C ) =
d s .
Then, the following holds.
(1) For any policy π linear in ¯ z h , a one-step future-dependent value function g πh ( · ) exists, and is linear in ψ h ( · ) .
(2) Letting d ψ h be the dimension of ψ h , we set G h
= { θ ⊤ ψ h ( · ) | θ ∈
R d ψh } and Π being linear in ¯ z h .
Then LQG satisfies Definition 3 with PO-bilinear rank at most
O ( { 1 + ( H − 1)( d o + d a ) + d s } 2 )
We have two remarks.
First, when π outt = δ ( a = 0) , K-step future-dependent value functions exist when [ C ⊤ , ( CA ) ⊤ , . . .
, ( CA K − 1 ) ⊤ ] is full raw rank.
This assumption is referred to as observability in control theory [28].
Secondly, the PO-bilinear rank scales polynomially with respect to H, d o , d a , d s even with M = H − 1 .
As we show in Section M, due to this fact, we can compete against the globally optimal policy with polynomial sample complexity.
4.3 Observable Hilbert Space Embedding POMDPs
We consider HSE-POMDPs that generalize tabular POMDPs and LQG.
Proofs here are deferred to
Section C.4.
Consider any h ∈
[ H ] .
Given a policy π h : ¯ Z h
→ A , we define the induced transition operator T π
= { T π ; h } Hh =1 as ( z h , s h +1 ) ∼ T π ; h ( z h − 1 , s h ) , where we have o h ∼ O
( s h ) , a h ∼ π h (¯ z h ) , s h +1 ∼ T ( s h , a h ) .
Namely, T π is the transition kernel of some Markov chain induced by the policy π .
The HSE-POMDP assumes two conditional distributions O ( ·| s ) and T π ( · , ·| z, s ) have conditional mean embeddings.
Example 3 (HSE-POMDPs) .
We introduce features ϕ h :
Z h − 1 × S → R d ϕh , ψ h : Z h − 1 ×
O
→ R d ψh .
We assume the existence of the conditional mean embedding operators: (1) there exists a matrix K h such that for all z ∈ Z h − 1 , s ∈ S , E o ∼ O
( ·| s ) ψ h ( z, o ) =
K h ϕ h ( z, s ) and (2) for all π ∈ Π , there exists a matrix T π ; h , such that E z h ,s h +1 ∼ T π ; h ( z h − 1 ,s h ) ϕ
h +
( z h , s h +1 )
=
T π ; h ϕ h ( z h − 1 , s h ) .
The existence of conditional mean embedding is a common assumption in prior RL works on learning dynamics of HMMs, PSRs, [64, 7] and Bellman complete linear MDPs [77, 18, 11, 26].
HSEPOMDPs naturally capture tabular POMDPs and LQG.
For tabular POMDPs, ψ h and ϕ h are one-hot encoding features.
In LQG, ϕ h and ψ h are quadratic features we define in Section 4.2.
Here for simplicity, we focus on finite-dimensional features ϕ h and ψ h .
Extension to infinite-dimensional
Reproducing kernel Hilbert Space is deferred to Section C.4.
The following shows the existence of future-dependent value functions and the PO-bilinear rank decomposition.
Lemma 4 (PO-bilinear rank of observable HSE-POMDPs) .
Assume K h is full column rank (observability), and V πh ( · ) is linear in ϕ h for any π ∈ Π ,
h ∈
[ H ] .
Then the following holds.
(1) A one-step future-dependent value function g π h ( · ) exists for any π ∈ Π ,
h ∈
[ H ] , and is linear in ψ h .
(2) We set a value function class G h
= { w ⊤ ψ h ( · ) | w ∈ R d ψh } , policy class Π h ⊂ [ ¯ Z h → ∆( A )] .
Then
HSE-POMDP satisfies Definition 3 with PO-bilinear rank at most max h ∈
[ H ] d ϕ h .
The first statement can be verified by noting that when V πh ( · ) = ⟨ θ h , ϕ h ( · ) ⟩ , future-dependent value functions take the following form g πh ( · ) = ⟨ ( K † h )
⊤ θ h ) , ψ h ( · ) ⟩ where we leverage the existence of the conditional mean embedding operator K h , and that K h is full column rank (thus K † h K h
=
I d ϕh ).
Note that the PO-bilinear rank depends only on the dimension of the features ϕ h without any explicit dependence on the length of memory.
Algorithm and Complexity
In this section, we first give our algorithm followed by a general sample complexity analysis.
We then instantiate our analysis to specific models considered in Section 4.
5.1 Algorithm
We first focus on the cases where models satisfy the PO-bilinear AC model (i.e., Definition 3) with finite action and with one-step future-dependent value function.
We present our algorithm P ROVABLE in Algorithm 1.
Note P ROVABLE is agnostic to the form of X h and W h .
Inside iteration t , given the latest learned policy π t , we define Bellman error for all pairs ( π, g ) where the Bellman error is averaged over the samples from π t .
Here, to evaluate the Bellman loss for any policy π ∈ Π , we use importance sampling by running U ( A ) rather than executing a policy π so that we can reuse samples.
2 A pair ( π, g ) that has a small total Bellman error intuitively means that given the data so far, g could still be a value future-dependent value function for the policy π .
Then in the constrained optimization formulation, we only focus on ( π, g ) pairs whose
Bellman errors are small so far.
Among these ( π, g ) pairs, we select the pair using the principle of optimism in the face of uncertainty.
We remark the algorithm leverages some design choices from the Bilinear-UCB algorithm for MDPs [16].
The key difference between our algorithm and the
Bilinear-UCB is that we leverage the actor-critic framework equipped with value future-dependent value functions to handle partially observability and agnostic learning.
Remark 3 (Three Important Extensions) .
By extending Algorithm 1, we can consider more general algorithms to include three important cases.
The first extension is the minimax version with discriminators to capture low-rank observable PMODPs and M-step decodable POMDPs.
The detail is in
Section P. Secondly, although algorithms so far implicitly assume the action is finite, we can consider LQG with continuous action by employing a G-optimal design over actions.
The detail is in Section D.2.
The third extension is the multi-step future case, which can capture overcomplete
POMDPs and general PSRs.
The discussion is deferred to Section D.
5.2 Sample Complexity
We show a sample complexity result by using reduction to supervised learning analysis.
We begin by stating the following assumption which is ensured by standard uniform convergence results.
Assumption 2 (Uniform Convergence) .
Fix h ∈
[ H ] .
Let D ′ h be a set of m i.i.d tuples following ( z h − 1 , s h , o h ) ∼ d π t h , a h ∼ U
( A ) , s h +1 ∼ T ( s h , a h ) , o h +
∼ O
( s h +1 ) .
With probability 1 − δ , sup π ∈ Π ,g ∈G | ( E D ′ h − E )[
[Equation] [Equation]
, δ )
For h = 1 , we also require sup g 1 ∈G 1 | E D ′
[ g 1 ( o 1 )] − E [ E D ′ 1 [ g 1 ( o 1 )]] | ≤ ϵ ini, 1 ( m, G , δ ) .
Remark 4 (Finite function classes) .
The term ϵ gen depends on the statistical complexities of the function classes Π , G .
As a simple example, we consider the case where Π and G are discrete.
In this case, we have ϵ gen,h ( m, Π , G , δ )
=
O ( (cid:112) ln( | Π ||G| /δ ) /m ) , and ϵ ini, 1 ( m, G , δ ) =
O ( (cid:112) ln( |G| /δ ) /m ) , which are standard statistical complexities for discrete function classes Π and G .
Achieving this result simply requires standard concentration and a union bound over all functions in Π , G .
Under Assumption 2, when the model is PO-bilinear with rank d , we get the following.
Theorem 1 (PAC guarantee of P ROVABLE ) .
Suppose we have a PO-bilinear AC class with rank d .
Suppose Assumption 2, sup π ∈ Π
∥ X h ( π ) ∥
≤ B X and sup π ∈ Π ,g ∈G ∥ W h ( π, g )
∥ ≤ B W for any h ∈
[ H ] .
By setting T = 2 Hd ln (cid:16) 4 Hd (cid:16) B 2 X B 2 W ˜ ϵ 2 gen + 1 (cid:17)(cid:17) , R = ϵ 2 gen where ϵ gen :=
[Equation]
gen,
[Equation] [Equation]
With probability at least 1 − δ , letting π ⋆ = argmax π ∈ Π J ( π ⋆ ) , we have
[Equation]
The total number of samples used in the algorithm is mTH .
Informally, when ϵ gen ≈ ˜ O (1 / √ m ) , to achieve ϵ -near optimality, the above theorem indicates that we just need to set m ≈ ˜ O
(1 /ϵ 2 ) , which results a sample complexity scaling ˜ O
(1 /ϵ 2 ) (since T only scales ˜ O ( dH ) ).
We give detailed derivation and examples in the next section.
5.3 Examples
Hereafter, we show the sample complexity result by using Theorem 1.
For complete results, refer to
Section I–N.
5.3.1 Finite Sample Classes
We consider the case where the hypothesis class is finite and admits PO-bilinear rank decomposition.
Example 4 (Finite Sample Classes) .
Consider the case when Π and G are finite and the PO-bilinear rank assumption is satisfied.
When Π and G are infinite hypothesis classes, |F| and |G| are replaced with their L ∞ -covering numbers, respectively.
Theorem 2 (Sample complexity for discrete Π and G (informal)) .
Let ∥G h ∥ ∞ ≤ C G , r h ∈ [0 , 1] for any h ∈
[ H ] and the PO-bilinear rank assumption holds with PO-biliear rank d .
By letting | Π max | = max h | Π h | , |G max | = max h |G h | , with probability 1 − δ , we can achieve J ( π ⋆ ) − J (ˆ π ) ≤ ϵ when we use samples
[Equation]
Here, Polylog( d, H, |A| , ln( |G| ) , ln( | Π | ) , ln(1 /δ ) , ln( B X ) , ln( B W ) , ln(1 /δ ) , 1 /ϵ ) are omitted.
5.3.2 Observable Undercomplete Tabular POMDPs
We start with tabular POMDPs.
The details here is deferred to Section K.
Example 1 (continuing from p. 6) .
In tabular models, recall the PO-bilinear rank is at most d = |O| M |A| M |S| .
We suppose r h ∈ [0 , 1] for any h ∈ [ H ] .
Assuming O is full-column rank, to satisfy the realizability, we set G h
= {⟨ θ, 1 ( z ) ⊗ O † 1 ( o ) ⟩ | ∥ θ ∥ ∞ ≤ H } where ∥ O † ∥ 1 ≤ 1 /σ 1 and 1 ( z ) , 1 ( o ) are one-hot encoding vectors over Z h − 1 and O , respectively.
We set Π h
=
[ ¯ Z h → ∆( A )] .
Then, the following holds.
Theorem 3 (Sample complexity for undercomplete tabular models (Informal)) .
With probability 1 − δ , we can achieve J ( π ⋆ ) − J (ˆ π ) ≤ ϵ when we use samples ˜ O (cid:0) |S| 2 |A| 3 M +3 |O| 3 M +1 H 6 (1 /ϵ ) 2 (1 /σ 1 ) 2 ln(1 /δ )
(cid:1) .
Here, polylog( |S| , |O| , |A| , H, 1 /σ 1 , ln(1 /δ )) are omitted.
Firstly, while the above error incurs |O| M |A| M , we will later see in Section G.2.2 when we use the more general definition of PO-bilinear AC class and combine a model-based perspective, we might be able to remove |O| M from the error bound.
The intuition here is that the statistical complexity still scales with |S| 2 |A| + |O||A| and does not incur |O| M .
At the same time, although PO-bilinear rank currently scales with |O| M ||A| M |S| , we can show that it can be just |S| with a more refined definition.
Secondly, ∥ O † ∥ 1 ≤ 1 /σ 1 can be replaced with other analogous conditions ∥ O † ∥ 2 ≤ 1 /σ 2 .
Here, note ∥ O † ∥ 1 = 1 / { min x ∥ O x ∥ 1 / ∥ x
∥ 1 } , ∥ O † ∥ 2 = 1 / { min x ∥
O x ∥ 2 / ∥ x ∥ 2 } .
The reason why we use 1 -norm is to invoke the result [23] to achieve the near global optimality as in the next paragraph.
Near global optimality.
Finally, we consider the PAC guarantee against the globally optimal policy.
As shown in [23], it is enough to set M =
O ((1 /σ 41 )
ln( SH/ϵ ))
to compete with the globally optimal policy π ⋆ gl .
Thus we achieve a quasipolynomial sample complexity when competing against π ⋆ gl .
Theorem 4 (Sample complexity for undercomplete tabular models (Informal) — competing against π ⋆ gl ) .
With probability 1 − δ , we can achieve J ( π ⋆ gl ) − J (ˆ π ) ≤ ϵ when we use samples at most poly( |S| , |A| ln( |S| H/ϵ ) /σ 41 , |O| ln( |S| H/ϵ ) /σ 41 , H, 1 /σ 1 , 1 /ϵ, ln(1 /δ )) .
Remark 5 (Overcomplete Tabular POMDPs) .
We can similarly consider the sample complexity of overcomplete POMDPs.
We would incur the additional |A| K .
The detail is in Section L.
5.3.3 Observable LQG
Now let us revisit LQG.
The detail here is deferred to Section M.
We show that P ROVABLE can compete against the globally optimal policy with polynomial sample complexity.
Example 2 (continuing from p. 6) .
In LQG, by setting H =
M − 1 , we achieve a polynomial sample complexity when competing against the globally optimal policy π ⋆ gl .
Theorem 5 (Sample complexity for LQG (informal) – competing against π ⋆ gl ) .
Consider a linear policy class Π h
= { δ ( a h =
¯
K
h ¯ z h ) | ∥ ¯ K h ∥ ≤ Θ } .
and assume max( ∥ A ∥ , ∥ B ∥ , ∥ C ∥ ,
∥ Q ∥ , ∥ R ∥ ) ≤ Θ and all policies induce a stable system (we formalize in Section M).
With probability 1 − δ , we can achieve J ( π ⋆ gl ) − J (ˆ π ) ≤ ϵ when we use samples at most poly( H, d s , d o , d a , Θ , ∥ C † ∥ , ln(1 /δ ))
× (1 /ϵ ) 2 .
5.3.4 Observable HSE-POMDPs
Next, we study HSE-POMDPs.
The details here is deferred to Section J.
Example 3 (continuing from p. 7) .
In HSE-POMDPs, PO-bilinear rank is at most max h d ϕ h .
Suppose ∥ ψ h ∥ ≤ 1 and V πh ( · ) = ⟨ θ πh , ϕ h ( · ) ⟩ such that ∥ θ πh ∥ ≤ Θ V for any h ∈
[ H ] .
Then, to satisfies the realizability, we set G h
= {⟨ θ, ψ h ( · ) ⟩ | ∥ θ ∥ ≤ Θ V /σ min ( K ) } where σ min ( K ) =
min h ∈
[ H ] 1 / ∥ K † h ∥ .
Theorem 6 (Sample complexity for HSE-POMDPs (Informal)) .
Let d ψ = max h
{ d ψ h } , d ϕ = max h { d ψ h } , | Π max | = max h ( | Π h | ) .
Suppose r h lies in [0 , 1] for any h ∈
[ H ] .
Then, with probability 1 − δ , we can achieve J ( π ⋆ ) − J (ˆ π ) ≤ ϵ when we use samples
[Equation]
Here, polylog( d ϕ , d ψ , |A| , Θ
[Equation]
, ln( |
[Equation]
max | ) ,
[Equation]
( K ) , 1 /ϵ, ln(1 /δ ) , σ max ( T ) , σ max ( K )) are omitted and σ max ( K ) =
max h ∈ [ H ] ∥
K h ∥ , σ max ( T ) = max π ∈ Π ,h ∈ [ H ] ∥ T π : h ∥ .
Note that the sample complexity above does not explicitly depend on the memory length M , instead it only explicitly depends on the dimension of the features ϕ, ψ .
In other words, if we have a feature mapping ψ h that can map the entire history (i.e., M = H ) to a low-dimensional vector (e.g., LQG), our algorithm can immediately compete against the global optimality π ⋆ gl .
5.4 PSRs, M -step decodable POMDPs and Low-rank Observable POMDPs
The result of PSRs is deferred to Section Section E. Besides, our generalized framework can capture two models: (1) M -step decodable POMDPs, and (2) observable POMDPs with the latent low-rank transition.
The discussion is deferred to Section P.
The summary of the results is stated in Section 1.
Summary
We propose a PO-bilinear actor-critic framework that is the first unified framework for provably efficient RL on large-scale partially observable dynamical systems.
Our framework can capture not only many models where provably efficient learning has been known such as tabular POMDPs,
LQG and M-step decodable POMDPs, but also models where provably efficient RL is not known such as HSE-POMDPs, PSRs, and low-rank observable POMDPs.
Our unified actor-critic based algorithm— P ROVABLE provably performs agnostic learning by searching for the best memory-based policy.
For special models such as observable tabular POMDPs, LQG, and low-rank POMDPs, by leveraging their special properties, i.e., the exponential stability of Bayesian filters in tabular and low-rank POMDPs, and existence of a compact featurization of histories in LQG, we are able to directly compete against the global optimality without paying an exponential dependence on horizon.
Acknowledgement
We thank Nan Jiang for the valuable discussions on PSRs.
MU and NK acknowledge funding support from NSF IIS-1846210 and Masason Foundation. WS acknowledges funding support from NSF IIS-2154711.
A Additional Related Works
B Supplement for Section
B.1 PO-bilinear Actor-critic Class with Multi Step Future
C Supplement for Section
C.1 Observable Undercomplete Tabular POMDPs
C.2 Observable Overcomplete Tabular POMDPs
C.3 Observable Linear Quadratic Gaussian
C.4 Observable HSE-POMDPs
D Supplement for Section
D.1 Algorithm with Multi-Step Future-Dependent Value Functions
D.2 Algorithm for LQG with Continuous Action
E Predictive State Representations
E.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.2 Definition of PSRs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.3 Existence of future-dependent value functions . . . . . . . . . . . . . . . . . . . .
E
4 PO-
Rank Decomoposition . . . .
F Generalization of PO
AC Class
G Examples for Generalized PO-Bilinear AC Class
G.1 M -step decodable POMDPs . . . . .
G.2 Observable POMDPs
G.2.
[Table]
Global
Optimality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
G.2.
Revisiting Observable Undercomplete Tabular POMDPs . . . . . . . . . .
G.2.
Revisiting Observable Overcomplete POMDPs
G.2.
4 Revisiting M -step Decodable Tabular POMDPs
H Proof of Theorem
I Sample Complexity for Finite Function Classes
J Sample Complexity in Observable HSE POMDPs
K Sample Complexity in Observable Undercomplete Tabular POMDPs
L Sample Complexity in Observable Overcomplete Tabular POMDPs
L.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
L.2 Detail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
M Sample Complexity in LQG
M.1 Existence of Link Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
M.2 PO-bilinear Rank Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . .
M.3 Uniform Convergence . .
M.4 Sample Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
N Sample Complexity in PSRs
N.1 Existence of Link Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
N.2 PO-bilinear Rank Decomposition . .
N.3 Sample Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
N.4 Most General Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
O Proof of Theorem
P Sample Complexity in M -step Decodable POMDPs
P.
PO-bilinear Rank Decomposition (Proof of Lemma 17) . . . . . . . . . . . . . . .
P.
Uniform Convergence . .
P.
Proof of Main Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Q Sample Complexity in Observable POMDPs with Latent Low-rank Transition
Q.1 Existence of Value Future-Dependent Value Functions . . . . . . . . . . . . . . .
Q.2 PO-bilinear Rank Decomposition (Proof of Lemma 18) . . . . . . . . . . . . . . .
R Exponential Stability for POMDPs with Low-rank Transition
S Auxiliary Lemmas
A Additional Related Works
Generalization and function approximation of RL in MDPs.
In Markovian environments, there is a growing literature that gives PAC bounds with function approximation under certain models.
Some of the representative models are linear MDPs [36, 76], block MDPs [17, 51, 78], and low-rank
MDPs [2, 71].
Several general frameworks in [33, 66, 35, 21, 16] characterize sufficient conditions for provably efficient RL.
Each above model is captured in these frameworks as a special case.
While our work builds on the bilinear/Bellman rank framework [16, 33], when we naïvely reduce POMDPs to MDPs, the bilinear/Bellman rank is Θ( A H ) .
These two frameworks are only shown applicable to reactive POMDPs where the optimal policy only depends on the latest observation.
However, this assumption makes the POMDP model very restricted.
System identification for uncontrolled partially observable systems.
There is a long line of work on system identification for uncontrolled partially observable systems, among which the spectral learning based methods are related to our work [72, 29, 63, 8, 25, 54, 7, 40, 27, 67].
Informally, these methods leverage the high-level idea that under some observability conditions, one can use the sufficient statistics of (possibly multi-step) future observations as a surrogate for the belief states, thus allowing the learning algorithms to ignore the latent state inference and completely rely on observable quantities.
Our approach shares a similar spirit in the sense that we use sufficient statistics of future observations to replace latent states, and our algorithm only relies on observable quantities.
The major difference is that these prior works only focus on passive system identification for uncontrolled systems, while we need to find a high-performance policy by actively interacting with the systems for information acquisition.
Reinforcement learning in PSRs.
PSRs [32, 45, 62, 8, 69] are models that generalize POMDPs.
PSRs also rely on the idea of using the sufficient statistics of multi-step future observations (i.e., predictive states) to serve as a summary of the history.
Prior works on RL for PSRs [8, 40, 15, 44, 30] do not address the problem of strategic exploration and operate under the assumption that a precollected diverse training dataset is given and the data collection policy is a blind policy (i.e., it does not depend on history of observations).
To our knowledge, the only existing PAC learning algorithm for PSRs is limited to a reactive PSR model where the optimal policy depends just on the latest observation [33].
Our framework captures standard PSRs models that are strictly more general than reactive PSRs.
Future-dependent value functions.
Analogue of future-dependent value functions (referred to as bridge functions) are used in the literature of causal inference (offline contextual bandits)
[50, 13, 12, 37, 49, 61, 75] and offline RL with unmeasured confounders [4, 59].
However, their settings are not standard POMDPs in the sense that their setting is a POMDP with unmeasured confounders following [68].
Our setting is a standard POMDP without unmeasured confounders.
Here, we emphasize that their setting does not capture our setting.
More specifically, by taking [59] as an example, they require that logged data is generated by policies that can depend on latent states but cannot depend on observable states.
Thus, their definition of future-dependent value functions (called as bridge functions) is not applicable to our setting since the data we use is clearly generated by policies that depend on observations.
Due to this difference, their setting prohibits us from using future observations, unlike our setting.
Finally, we stress that our work is online, while their setting is offline.
Hence, they do not discuss any methods for exploration.
B Supplement for Section
We generalize Definition 3 to capture more models.
The first extension is to use multi-step futuredependent value functions.
This extension is essential to capture overcomplete POMDPs and multi-step PSRs.
B.1 PO-bilinear Actor-critic Class with Multi Step Future
In this section
[Equation]
we provide an extension to Definition 3 to incorporate multiple-step futures (i
[Equation]
e.,
K > 1 ).
For simplicity, we assume that π out =
U ( A )
[Equation]
The definition is then as follows.
The main difference is that we roll out a policy U ( A ) , K − 1 times to incorporate multi-step future-dependent value functions.
We introduce the notation
[Equation]
Then, combining the Bellman equation for state-value functions and the definition of K-step futuredependent value functions
[Equation]
we have
[Equation]
Thus, by taking expectations further with respect to ( z h − 1 , s h ) (i
[Equation]
e., z h − 1 , s h can be sampled from some roll-in policy)
[Equation]
we have
[Equation]
Hence, the Bellman loss of a pair ( π, g ) under a roll-in π ′ denoted by Br h ( π, g ; π ′ ) at h ∈ [ H ] is defined as
Br
[Equation]
The above is a proper loss function when we use multi-step futures.
Here is the structure we need for
Br h ( π, g ; π ′ ) .
Definition 4 (PO-bilinear AC Class for POMDPs with multi-step future) .
The model is a PO-bilinear class of rank d if G is realizable (regarding general K-step future-dependent value functions), and there exists W h : Π × G → R d and X h : Π → R d such that for all π ′ , π ∈ Π , g ∈ G and h ∈
[ H ] , 1.
We have:
[Equation]
a 1: h
[Equation]
1 ∼
[Equation]
W h
( π, g π ) =
for any π ∈ Π and the corresponding future-dependent value function g π in G .
We define d as the PO-bilinear rank.
C Supplement for Section
C
.1 Observable Undercomplete Tabular POMDPs
We need to prove Lemma 2.
In the tabular case, by setting
[Equation]
where 1 ( z ) , 1 ( o ) , 1 ( s ) are one-hot encoding vectors over Z h − 1 , O , S , respectively.
Then, we can regard the tabular model as an HSE-POMDP.
We can just invoke Lemma 4.
C.2 Observable Overcomplete Tabular POMDPs
We consider overcomplete POMDPs with multi-step futures.
The proofs are deferred to Section
C.2.
We have the following theorem.
This is a generalization of Lemma 1, i.e., when K = 1 , it is
Lemma 1.
Lemma 5.
Define a |T K | × |S| -dimensional matrix O K whose entry indexed by ( o h : h + K − 1 , a h : h + K − 2 )
∈
T K and s h ∈ S is equal to P ( o h : h + K − 1 , a h : h + K − 2 | s h ; a h : h + K − 2 ∼ U ( A )) .
When this matrix is full-column rank, K-step future-dependent value functions with respect to
U ( A ) exist.
Note a sufficient condition to satisfy the above is that a matrix O K ( a ′ h : h + K − 2 ) ∈ R |O|
K ×|S| whose entry indexed by o h : h + K − 1 ∈
O K and s h ∈ S is equal to P ( o h : h + K − 1 | s h ; a h : h + K − 2 = a ′ h : h + K − 2 ) is full-column rank for certain a ′ h :
h + K − 2 ∈
A K − 1 .
It says there is (unknown) action sequence with length K that retains information about latent states.
We next calculate the PO-bilinear rank.
Importantly, this does not depend on |A| K and |O| K .
Lemma 6.
Set a future-dependent value function class G h
=
[ ¯ Z K → [0 , C G ]] for certain C G ∈ R + and a policy class Π h
=
[ ¯ Z h → ∆( A )] .
Then, the model satisfies PO-bilinear rank condition with
PO-bilinear rank (Definition 4) at most |S| ( |O||A| ) M .
Note that the bilinear rank is still |S| ( |O||A| )
M (just |S| in the more general definition in Section F).
Crucially, it does not depend on the length of futures K .
Proof of Lemma 5 Consider any function f : Z h − 1 × S → R (thus, this captures all possible V πh ).
Denote 1 ( z ) as the one-hot encoding of Z h − 1 (similarly for 1 ( s ) over S and 1 ( t ) over T K ).
We have f ( z, s ) =
⟨ f, 1 ( z ) ⊗ 1 ( s ) ⟩ = ⟨ f, 1 ( z ) ⊗ (( O K ) † O K 1 ( s )) ⟩ , where we use the assumption that rank( O K ) = |S| and thus ( O K ) † O K
= I .
Then,
[Equation]
which means that the value bridge function corresponding to f ( · ) is
[Equation]
■
Proof of Lemma 6 Recall we want to show the low-rank property of the following loss function:
[Equation]
We consider an expectation conditioning on z h − 1 and s h .
For some vector θ π,g ∈ R |Z h − 1 |×|S| , which depends on π , we write it in the form of ⟨ θ π,g , 1 ( z h − 1 , s h ) ⟩ where 1 ( z h − 1 , s h ) is the one-hot encoding vector over Z h − 1 × S .
Then, the loss for ( π, g ) is equal to ⟨ θ π,g , E [ 1 ( z h − 1 , s h ); a 1: h − 1 ∼ π ′ ] ⟩ .
Hence, we can take X ( π ′ )
=
E [ 1 ( z h − 1 , s h ); a 1: h − 1 ∼ π ′ ] and W ( π ) = θ π,g .
■
C.3 Observable Linear Quadratic Gaussian
We need to prove Lemma 3.
The proof is further deferred to Section M.
C.4 Observable HSE-POMDPs
We first provide the proof of Lemma 4.
Then, we briefly mention how we extend to the infinitedimensional setting.
Proof of the first statement in Lemma 4 First, we need to show value bridge functions exist.
This is proved noting
[Equation]
Thus
[Equation]
⟨ ( K † h )
⊤ θ πh , ψ h (¯ z h ) ⟩ is a value bridge function.
■
Proof of the second statement in Lemma 4 Consider a triple ( π ′ , π, g ) ∈
Π × Π × G , with g h ( · ) =
θ ⊤ h ψ h ( · ) and g πh = ⟨ θ ⋆h , ψ h ( · ) ⟩ , we have:
[Equation]
which verifies the bilinear structure, i.e., X
[Equation]
( π ′ )
=
E [ ϕ h ( z h − 1 , s h ); a 1: h − 1 ∼ π ′ ] , and W h ( π, g ) =
K ⊤ h ( θ h − θ ⋆h ) − T ⊤ π ; h K ⊤ h
+1 ( θ h +1 − θ ⋆h +1 ) , and shows that the bilinear rank is at most max h d ψ h .
■
Infinite dimensional HSE-POMDPs Consider the case ϕ h and ψ h are features in infinite dimensional RKHS.
By assuming that the spectrum of the operator K h is decaying with a certain order, we can still ensure the existence of value bridge functions even if d ϕ h and d ′ ψ h are infinite dimensional.
Next, we consider the PO-bilinear rank.
We can still use the decomposition in the proof above.
While the PO-bilinear rank itself in the current definition is infinite-dimensional, when we get the PAC result later, the dependence on the PO-bilinear rank comes from the information gain based on X h ( π ) , which is the intrinsic dimension of X h ( π ) .
Thus, we can easily get the sample complexity result by replacing d ψ h with the information gain over ψ h ( · ) [65].
Generally, to take infinite dimensional models into account, the PO-bilinear rank in Definition 3 can be generalized using the critical information gain [16].
D Supplement for Section
In this section, we first consider the case with multi-step futures.
Next, we present a modification to handle LQG with continuous action in Definition 3.
D.1 Algorithm with Multi-Step Future-Dependent Value Functions
Finally, we consider the case with multi-step futures in Algorithm 2 when π out =
U ( A ) .
Recall the notation ¯ z
Kh =
( z h − 1 , o h : h + K − 1 , a h : h + K − 2 ) .
The only difference is in the process of data collection.
Particularly, at every iteration t , we roll-in using π t to (and include) time step h − 1 , we then roll-out by switching to U ( A ) for K steps.
D.2 Algorithm for LQG with Continuous Action
Our algorithm so far samples a h from U ( A ) and performs importance weighting in designing the loss σ th , which will incur a polynomial dependence on |A| as we will see in the next section.
However, among the examples that we consider in Section 4, LQG has continuous action.
If we naïvely sample a h from a ball in R d a and perform (nonparametric) importance weighting, we will pay exp( d a ) in our sample complexity bound, which is not ideal for high-dimension control problems.
To avoid exponential dependence on d a , here we replace U ( A ) with a d -optimal design over the action’s quadratic feature space.
Here, we want to evaluate the Bellman error of ( π, g ) pair under a roll-in policy π ′ :
[Equation]
where u h (¯ z h , a h , r h , o h +1 ; θ )
=
θ ⊤ h ψ (¯ z h ) − r h
( s h , a h ) − θ ⊤ h +1 ψ h +
(¯ z h +1 ) for any linear deterministic policy π ∈ Π (here g h ( · ) :
= θ ⊤ h ψ ( · ) ) using a single policy.
In other words, we would like to get
1: Input: Future-dependent value function class G = {G h } , G h ⊂ [ ¯ Z Kh → R ] , Policy class Π
= { Π h } , Π h ⊂ [ ¯ Z h → ∆( A )] , parameters m ∈ N , R ∈ R 2: Define
[Equation]
3: Initialize π 0 ∈ Π 4: Form the first step dataset D
= { ¯ z K ; i 1 } mi =1 where each ¯ z K is generated by following a 1: K − 1 ∼
U ( A ) in an i.i.d manner.
5: for t =
→ T − 1 do 6: for any h ∈
[ H ] , define the Bellman error
[Equation]
where D th means empirical approximation by executing a 1: h − 1 ∼ π t , a h : h + K − 1 ∼ U ( A ) and collecting m i.i.d tuples.
Select policy optimistically as follows (here note g = { g h } Hh =1 ) (
[Equation]
t +1 , g t +
[Equation]
:= argmax
[Equation]
8: end for 9: Output: Randomly choose ˆ π from ( π 1 , · · · , π T ) .
a good loss l h such that
[Equation]
∼ π ′ , a h ∼ π e ] for some policy π e without incuring exponential dependence on d a .
We explain how to design such a loss function l h ( · ; π, g ) step by step.
First Step The first step is to consider the conditional expectation on (¯ z h , s h , a h ) .
Here, using the quadratic form of ψ , we can show that there are some c 0 : ¯ Z h × S → R , c 1 : ¯ Z h × S →
R ( d a
[Equation]
d s + d ¯ zh ) 2 , c
[Equation]
R :
[Equation]
where κ ( a ) =
[ a ⊤ , ( a ⊗ a ) ⊤ ] ⊤ .
Then, the Bellman loss we want to evaluate can be written in the form of
[Equation]
Second step The second step is to compute a d-optimal design for the set { κ ( a ) : a ∈ R d a , ∥ a ∥ 2 ≤
Z } for certain enough large Z ∈ R , and denote a 1 , . .
. , a d ⋄ as the supports on the d-optimal design.
Note in LQG, though we cannot ensure the action lives in the compact set, we can still ensure that in high probability and it suffices in our setting as we will see.
Since the dimension of k ( a ) is d a + d 2 a , we can ensure d ⋄ ≤ ( d a + d 2 a )( d a + d 2 a + 1) / 2 [43, 39].
Here is a concrete theorem we invoke.
Theorem 7 (Property of G-optimal design) .
Suppose X ∈ R
d is a compact set.
There exists a distribution ρ over X such that:
• ρ is supported on at most d ( d + 1) / 2 points.
• For any x ′ ∈ X , we have x ′⊤ E x ∼ ρ
[ xx ⊤ ] − 1 x ′ ≤ d .
We have the following handy lemma stating any κ ( a ) is spanned by { κ ( a i ) }
d ⋄ i =1 .
Lemma 7.
Let K =
[Equation]
[Equation] [Equation]
a
[Equation] [Equation]
a
[Equation]
) , · · · , ρ 1 / 2 ( a d ⋄ ) κ ( a d ⋄ )] and α
[Equation]
a ) =
K ⊤ ( KK ⊤ ) − 1 k ( a ) .
Then, it satisfies
[Equation]
Proof.
Since K is full-raw rank from the construction of G-optimal design, KK ⊤ is invertible.
Then, we have
[Equation]
For the latter statement, we have
[Equation]
We use a property of G-optimal design in Theorem 7.
For the last statement, we have
[Equation]
from CS inequality.
Third Step The third step is combining current facts.
Recall we want to evaluate
E [ u
[Equation] [Equation]
z
[Equation]
[Equation] [Equation]
, o
[Equation]
In addition, the following also holds:
[Equation]
o h +1 ; θ ); a 1: h − 1 ∼ π ′ , a h ∼ do ( a i )]
=
E [
[Equation] [Equation]
s
[Equation]
[Equation] [Equation]
, s
[Equation]
θ ) κ ( a i )
[Equation]
a 1: h − 1 ∼ π ′ ]
E [ u h (¯ z h , a h , r h , o h
+1 ; θ ); a 1: h − 1 ∼ π ′ , a h ∼ do (0)]
=
E [ c 0 (¯ z h , s h ; θ ); a 1: h − 1 ∼ π ′ ]
Here, we use κ (0) = 0 .
This concludes that
[Equation]
=
E (cid:34)(cid:32) 1 − d
[Equation]
(cid:88) i
= α
i ( π (¯ z h ))
(cid:33) u
[Equation]
Thus, we can perform policy evaluation for a policy π if we can do intervention from do (0) , do ( a 1 ) , · · · , do ( a d ⋄ ) .
Fourth Step The fourth step is replacing do (0) , do ( a 1 ) , · · · , do ( a d ⋄ ) with a single policy that uniformly randomly select actions from the set { 0 , a 1 , . .
. , a d ⋄ } , which we denote as a ∼ U (1 + d ⋄ ) .
Using importance weighting, we define the loss function for π, θ as follows:
[Equation]
where U (1 + d ⋄ ) is a uniform action over 0 , a 1 , · · · , a d ⋄ and f h (¯ z h , a h , r h , o h
+1 ; θ, π ) =
| 1 + d ⋄ | (cid:32)
I ( a h = 0) (cid:32) 1 − d ⋄ (cid:88) i = α i ( π (¯ z h )) (cid:33) + d ⋄ (cid:88) i =
I ( a h = a i ) α i ( π (¯ z h )) (cid:33) u h (¯ z h , a h , r h , o h +1 ; θ ) .
The term 3is equal to Br h ( π, g ; π ′ )
we want to evaluate.
Summary To summarize, we just need to use the following loss function in line 6 in Algorithm 1:
[Equation]
where l h (¯ z h , a h , r h , o h +1 ; θ, π ) is
[Equation]
and D th is a set of m i.i.d samples following the distribution induced by executing a 1: h − 1 ∼ π ′ , a h ∼
U (1 + d ⋄ ) .
Values Z 1 , Z 2 , Z 3 in indicators functions are some large values selected properly later.
Due to unbounded Gaussian noises in LQG, indicators functions for truncation is introduced here for technical reason to get valid concentration in Assumption 2.
E Predictive State Representations
We first give a summary of our results in PSRs.
Then, we first add several discussions to explain core tests in detail.
Next, we show the existence and form of future-dependent value functions.
Finally, we calculate the PO-bilinear rank.
In this section, we will focus on the general case where tests could be multiple steps.
E.1 Summary
In this section, we demonstrate that our definition and algorithm applies to PSRs — models that strictly generalize POMDPs [45, 62].
Below, we first briefly introduce PSRs, followed by showing that it is a PO-bilinear AC model.
Throughout this section, we will focus on discrete linear PSRs.
We also suppose reward at h is deterministic function of ( o h , a h ) conditional on τ ah +1 where τ ah = ( o 1 , a 1 , · · · , o h − 1 , a h − 1 ) .
Given τ ah , the dynamical system generates o h ∼ P ( ·| τ ah ) .
Here we use the superscript a on τ ah to emphasize that the τ ah ends with the action a h − 1 .
PSRs use the concept of test , which is a sequence of future observations and actions, i.e., for some test t =
( o h : h + W − 1 , a h : h + W − 2 ) with length W ∈ N + , we define the probability of test t being successful P ( t | τ ah ) as P ( t | τ ah ) :
= P ( o h : h + W − 1 | τ ah ; do ( a h : h + W − 2 )) which is the probability of observing o h : h + W − 1 by actively executing actions a h : h + W − 2 conditioned on history τ ah .
We now explain one-step observable PSRs while deferring the general multi-step observable setting to Section E. A one-step observable PSR uses the observations in O as tests, i.e., tests with length 1.
Definition 5 (Core test set and linear PSRs) .
A core test set T ⊂ O contains a finite number of tests (i.e., observations from O ).
For any h , any history τ ah , any future test t h =
( o h : h + W − 1 , a h : h + W − 2 ) for any W ∈ N + , there exists a vector m t h ∈ R |T | , such that the probability of t h succeeds conditioned on τ ah can be expressed as: P ( t h | τ ah )
= m
⊤ t h [ P ( o | τ ah )]
o ∈T , where we denote q τ ah :=
[ P ( o | τ ah )]
o ∈T as a vector in R |T | with entries equal to P ( o | τ ah ) for o ∈ T .
The vector q τ ah is called predictive state.
A core test set T that has the smallest number of tests is called a minimum core test set denoted as Q .
PSRs are strictly more expressive than POMDPs in that all POMDPs can be embedded into PSRs whose size of the minimum core tests is at most |S| ; however, vice versa does not hold [45].
For example, in observable undercomplete POMDPs (i.e., O full column rank) , the observation set O can serve as a core test set, but the minimum core test set Q will have size |S| .
Here, we assume we know a core test set T that contains Q ; however, we are agnostic to which set is the actual Q .
In the literature on PSRs, this setting is often referred to as transform PSRs [8, 57].
Now we define a future-dependent value function in PSRs.
First, given an M -memory policy, define
V πh ( τ ah ) =
E [ (cid:80)
Ht
= h r t | τ ah ; a h : H ∼ π ] , i.e., the expected total reward under π , conditioned on the history τ ah .
Note that our value function here depends on the entire history.
Definition 6 (General future-dependent value functions) .
Consider an M -memory policy π .
One-step general future-dependent value functions g π h : Z h − 1 × T → R at step h ∈
[ H ] are defined as solutions to
[Equation]
This definition is more general than Definition 3 since (4) implies (1) in POMDPs by setting O = T .
In PSRs, we can show the existence of this general future-dependent value function.
Lemma 8 (The existence of future-dependent value functions for PSRs) .
Suppose T is a core test set.
Then, a one-step future-dependent value function g πh always exists.
The high-level derivation is as follows.
Using the linear PSR property, one can first show that V πh ( τ ah ) has a bilinear form V πh ( τ ah )
= 1 ( z h − 1 )
⊤ J πh q τ ah , where 1 ( z ) ∈ R |Z h − 1 | denotes the one-hot encoding vector over Z h − 1 , and J πh is a |Z h − 1 | × |T | matrix.
Then, given any τ ah and o ∼ P ( ·| τ ah ) , for some |Z h − 1 | × |T | matrix J , we can show g h ( z h − 1 , o ) :
= 1 ( z h − 1 )
⊤ J [ 1 ( t = o )]
t ∈T satisfies the above, where [ 1 ( t = o )] t ∈T ∈ R |T | is a one-hot encoding vector over T and serves as an unbiased estimate of q τ ah .
Finally, we show that PSR admits PO-bilinear rank decomposition (Definition 3).
Lemma 9.
Suppose a core test set T includes a minimum core test set Q .
Set Π h
=
[ ¯ Z h → ∆( A )] and G h
= { ( z h − 1 , o ) (cid:55)→ 1 ( z h − 1 )
⊤ J [ 1 ( t = o )]
t ∈T | J ∈ R |Z
h − 1 |×|T | } , the PO-bilinear rank is at most ( |O||A| ) M |Q| .
Then, Algorithm 1 is directly applicable to PSRs.
Note that here the PO-bilinear rank, fortunately, rank scales with |Q| but not |T | .
The dependence ( |O||A| )
M comes from the dimension of the “feature" of memory 1 ( z h − 1 ) .
If one has a compact feature representation ϕ : Z h − 1 → R d , such that V πh ( τ ah ) =
ϕ
( z h − 1 )
⊤ J πh q τ a h is linear with respect to feature ϕ
( z h − 1 ) , then the PO-bilinear rank is d |Q| .
This implies that if one has a compact featurization of memory a priori, one can avoid exponential dependence on M .
Sample complexity.
We finally briefly mention the sample complexity result.
The detail is deferred to Section N.
The sample complexity to satisfy J ( π ⋆ ) − J (ˆ π ) ≤ ϵ is given as
[Equation]
where Θ W and Θ are some parameters associated with PSRs.
Here, there is no explicit dependence on |T | .
Note that in the worst case, ln |G max | scales as O ( |Z h − 1 ||T | ) , and ln | Π max | scales as
O ( |Z h − 1 ||O||A| ) .
E.2 Definition of PSRs
We first define core tests and predictive states [45, 62].
This definition is a generalization of
Definition 5 with multi-step futures.
We slighly abuse notation and denote τ ah := ( o 1 , a 1 , . .
, o h − 1 , a h − 1 ) throughout this whole section — note that τ ah here does not include o h .
Definition 7 (Core test sets and PSRs) .
A set T ⊂ ∪ C ∈ N + O C × A C − 1 is called a core test set if for any h ∈
[ H ] , W ∈ N + , any possible future (i.e., test)
t h =
( o h : h + W − 1 , a h : h + W − 2 ) ∈
O W ×A W −and any history τ ah , there exists m t h ∈ R |T | such that
[Equation]
The vector [ P ( t | τ ah )] t ∈T ∈ R |T | is referred to as the predictive state.
We often denote q τ ah =
[ P ( t | τ ah )]
t ∈T h .
To understand the above definition, we revisit observable undercomplete POMDPs and overcomplete POMDPs.
Example 1 (Observable undercomplete POMDPs) .
In undercomplete POMDPs, when O is fullcolumn rank, O is a core test.
Recall O is a matrix in R |O|×|S| whose entry indexed by o i ∈
O , s j ∈ S is equal to O
( o i | s j ) .
Lemma 10 (Core tests in undercomplete POMDPs) .
When O is full-column rank, O is a core test set.
Proof.
Consider any h ∈
[ H ] .
Given a |S| -dimensional belief state s τ ah =
[ P ( · | τ ah )] |S| with each entry P ( s h | τ ah ) , for any future t
=
( o h : h + W , a h : h + W − 1 ) , there exists a |S| -dimensional vector m
′ t such that P
o h : h + W | τ ah ; do ( a h : h + W − 1 ))
= ⟨ m ′ t , s τ ah ⟩ .
More specifically, m ′ t can be written as:
[Equation]
where O ( o |· ) ∈ R |S| is a vector with the entry indexed by s equal to O ( o | s ) , T a ∈ R |S|×|S| is a matrix with the entry indexed by ( s, s ′ ) equal to T ( s ′ | s, a h ) .
Here, note given a vector C , diag( C ) is define as a | C | × | C | diagonal matrix where the diagonal element corresponds to C .
Thus, we have
P ( o h : h + W | τ ah ; do ( a h : h + W − 1 ))
= ⟨ m ′ t , s τ ah ⟩ = ⟨ m ′ t , O † q τ ah ⟩ = ⟨ ( O † ) ⊤ m ′ t , q τ ah ⟩ , where q τ ah ∈ R |O| and
q τ ah ( o ) = P ( o | τ ah ) .
This concludes the proof.
Example 2 (Overcomplete POMDPs) .
We consider overcomplete POMDPs so that we can permit |S| ≥ |O| .
Lemma 11 (Core tests in overcomplete POMDPs) .
Recall T K
= O ×
( O × A ) K − 1 .
Define a |T K | × |S| -dimensional matrix O K whose entry indexed by ( o h : h + K − 1 , a h : h + K − 2 )
∈
T K , s h ∈ S is equal to P ( o h : h + K − 1 , a h : h
+ K − 2 | s h ; a h : h + K − 2 ∼ U ( A )) .
When this matrix is full-colmun rank for all h , T K is a core test set.
Proof.
Fix a test t =
( o ′ h :
h + K − 1 , a ′ h : h + K − 2 ) and consider a step
h ∈
[ H ] .
Then,
P ( o ′
[Equation]
, a ′
[Equation] [Equation]
−
[Equation]
s
[Equation]
; a
[Equation]
: h + K − 2 ∼ U ( A )) =
[Equation] [Equation]
[Equation] [Equation]
a h : h + K − 2 ∼ U ( A ))]
=
E [(1 / |A| K − 1 ) 1 ( o h : h
+ K − 1 = o ′ h :
h + K − 1 , a h : h + K − 2 = a ′ h :
h
+ K − 2 ) | s h ; a h : h + K − 2 ∼ do ( a ′ h : h + K − 2 )]
=
E [(1 / |A| K − 1 ) 1 ( o h : h
+ K − 1 = o ′ h : h
+ K − 1 ) | s h ; a h : h + K − 2 ∼ do ( a ′ h : h + K − 2 )]
= (1 / |A| K − 1 ) P ( o ′ h :
h + K − 1 | s h ; do ( a ′ h : h + K − 2 )) .
Thus, the assumption that T K is full column rank implies that that the matrix ¯ J h ∈ R |T K |×|S| with the entry indexed by ( t, s h ) being equal to P ( o ′ h : h + K − 1 | s h ; do ( a ′ h : h
+ K − 2 )) is full-column rank.
Define a |T K | -dimensional state q τ ah =
[ P ( t | τ ah )] t ∈T K given history τ ah .
By definition, we have
[Equation]
Using ¯ J h is full-column rank, we have s τ ah =
¯ M † h q τ ah .
Thus, using the format of m ′ t from the proof of Lemma 10, we can conclude that for any test t =
( o h : h + W , a h : h + W − 1 ) , we have
P ( o h : h + W | τ ; do ( a h : h + W − 1 ))
= ⟨ ( ¯ M † h ) ⊤ m ′ t , q τ ah ⟩ .
Thus, this concludes T K is a core test set.
Finally, we present an important property of predictive states, which corresponds to the Bayesian filter in POMDP.
Lemma 12 (Forward dynamics of predictive states) .
We have
[Equation]
When we define M o,a ∈ R |T |×|T | where rows are m o,a,t for t ∈ T , we can express the forward update rule of predictive states as follows:
[Equation]
Proof.
The proof is an application of Bayes’s rule.
We denote the observation part of t by t O and the action part of t A , respectively.
We have
[Equation]
This concludes the proof.
To further understand that why PSR generalizes POMDP, let us re-visit the undercomplete POMDPs (i.e., O being full column rank) again.
Set T
= O .
As we see in the proof of Lemma 10, the belief state s τ ∈ ∆( S ) together with O defines predictive state, i.e., q τ ah =
O s τ ah , with M o,a =
OT a diag( O ( o |· )) O † , and m ⊤ o =
⊤ diag( O ( o |· ))
O † .
Note that in POMDPs, matrix M o,a and vector m o all contain non-negative entries.
On other hand, in PSRs, M a,o and m a,o could contain negative entries.
This is the intuitive reason why PSRs are more expressive than POMDPs [45].
For the formal instance of a finite-dimensional PSR which cannot be expressed as a finite-dimensional POMDP, refer to [62, 31].
E.3 Existence of future-dependent value functions
We discuss the existence and the form of future-dependent value functions.
First, we define general future-dependent value functions with multi-step futures.
For notational simplicity, we assume here that the tests t ∈ T have the same length, i.e., there is a K ∈ N + , such that T ⊂
O K × A K − 1 .
Definition 8 (General future-dependent value functions in dynamical systems) .
Recall T ⊂
O K ×
A K − 1 is the set of tests.
At time step h , general future-dependent value functions g πh : Z h − 1 ×
O K A K − 1 → R are defined as solutions to the following:
[Equation]
where ρ out is some distribution over the action set T A induced by the test set, i.e., { t A : t ∈ T } .
Here, for t =
( o h : h + K − 1 , a h : h + K − 2 ) , we often denote o h : h + K − 1 and a h : h + K − 2 by t O and t A , respectively.
To show the existence of general future-dependent value functions for PSRs, we first study the format of value functions in PSRs.
The following lemma states that value functions for M -memory policies have bilinear forms.
Lemma 13 (Bilinear form of value functions for M -memory policies) .
Let ϕ ( · ) ∈ R |Z h − 1 | be a one-hot encoding vector over Z h − 1 .
Suppose T is a core test set.
Then, for any M -memory policy π , there exists J πh ∈ R |Z h − 1 |×|T | such that
[Equation]
Proof.
From Lemma 12, there exists a matrix M o,a ∈ R |T |×|T | such that via Bayes rule:
[Equation]
We use induction to prove the claim.
Here, the base argument clearly holds.
Thus, we assume
[Equation]
We have
[Equation]
a function of o h , a h conditional on ( τ a h , o h , a h ) .
We first check the first term (a) that contains rewards.
Using the fact that P ( o | τ ah )
= m
⊤ o q τ ah , this is equal to (cid:88)
[Equation]
Thus, it has a bilinear form, i.e., there exists some matrix J π 1 such that
[Equation]
where J π 1 is a matrix whose row indexed by z h − 1 is equal to (cid:80)
[Equation]
,a m ⊤
[Equation]
π h ( a | o, z
h − 1 ) r
( o, a ) .
Next, we see the second term (b).
Using (6), the second term is equal to (cid:88)
[Equation]
where we use the notation z h − 1 ⊕ o, a to represent the operation of appending ( o, a ) pair to the memory while maintaining the proper length of the memory by truncating away the oldest observationaction pair.
Thus, it has an again bilinear form ϕ ( z h − 1 )
⊤ J π 2 q τ ah and the matrix J π 2 can be defined such that its row indexed by z h − 1 is equal to (cid:80) o,a π h ( a | o,
z h − 1 ) ϕ ⊤ ( z h − 1 ⊕ o, a ) M πh +1 M a,o .
This concludes the proof.
Next, we check sufficient conditions to ensure the existence of general K-step future-dependent value functions.
Given T , we define the corresponding set of action sequences T A as T A :=
{ t A : t ∈ T } .
We set ρ out in (5) to be a uniform distribution over the set T A denoted by U ( T A ) .
Namely, U ( T A ) will uniformly randomly select a sequence of test actions from T A .
Lemma 14 (Existence of future-dependent value functions in PSRs) .
Suppose T is a core test.
There exists g πh : Z h − 1 × T such that
[Equation]
Proof.
We mainly need to design an unbiased estimator of the predictive state q τ
ah .
We use importance weighting to do that.
Given a h : h + K − 2 ∼ U
( T A ) , and the resulting corresponding random observations o h : h + K − 1 , we define the following estimator ˆ q τ ah ( o h : h + K − 1 , a h : h + K − 2 ) ∈ R |T | , such that its entry indexed by a test t ∈ T is equal to:
[Equation]
We can verify that
[Equation]
Then,
E [ˆ
[Equation]
ah .
With this estimator, now we can define the future-dependent value function using the bilinear form of
V πh ( τ ) , i.e.,
[Equation]
Using the fact that ˆ q τ ah ( o h : h + K − 1 , a h : h + K − 2 ) is an unbiased estimate of q τ ah , we can conclude the proof.
Since PSR models capture POMDP models, our above result directly implies the existence of the future-dependent value functions in observable POMDPs as well by using obtained facts in Exampleand 2 .
E.4 PO-Bilinear Rank Decomoposition
Finally, we calculate the PO-bilinear rank.
Here,
[Equation]
The Bellman error for ( g, π ) under a roll-in π ′ denoted by Br h ( g, π ; π ′ ) is defined as
[Equation]
In fact, Br h ( g, π ; π ′ ) =
0 for any general future-dependent value functions g π .
Our goal is to design a loss function l h ( · ) such that we can estimate the above Bellman error
Br h ( g, π ; π ′ ) using data from a single policy.
To do that, we design the following randomized action selection strategy.
Given a action sequence t A from a test t , let us denote ¯ t
A as a copy of t A but starting from the second action of t A , i.e., if t A = { a 1 , a 2 , a 3 } , then ¯ t
A = { a 2 , a 3 } .
Denote ¯ T A =
{ ¯ t A : t ∈ T } .
Our random action selection strategy first selects a h ∼ U
( A ) uniformly randomly from A , and then select a sequence of actions ¯ a uniformly randomly from T A ∪ ¯ T A .
Here, we remark the length of outputs is not fixed (i.e., ¯ a ∈ T A has length larger than the ¯ a ∈ ¯ T A ).
As a first step, we define two unbiased estimators for q τ ah and q τ ah +1 .
Conditioning on history τ ah , given actions a h ∼ U
( A ) followed by action sequence ¯
a h +1 ∼ U
( T A ∪ ¯ T A ) , denote the corresponding observations as o h , o h +1 , . . .
o h
+ | ¯ a h
+
| +1 .
We construct unbiased estimators for q τ ah and q τ ah +1 as follows.
As an unbiased estimator of q τ ah , we define ˆ q τ ah with the entry indexed by test t ′ ∈ T as follows:
[Equation]
Similarly, as an unbiased estimator of q τ ah +1 , we define ˆ q τ ah +1 with the entry indexed by test t ′ ∈ T as follows:
[Equation]
We remark the length of ¯ a in (7) and the one of (8) are different.
Then, by using importance sampling, we can verify
[Equation]
With the above setup, we can construct the loss function l for estimating the Bellman error.
We set the loss as follows:
[Equation]
Since we have shown that ˆ q τ ah and ˆ q τ ah +1 are unbiased estimators of q τ and q τ ah +1 , respectively, we can show that for any roll-in policy π ′ :
Br h ( π, g ; π ′ )
[Equation]
U ( T A ∪ ¯ T A ) (cid:3) .
The above shows that we can use l h ( · ) as a loss function.
Summary We can use the almost similar algorithm as Algorithm 1.
The sole difference is we need to replace σ th ( π, g ) with
[Equation]
where D th is an empirical approximation when executing a 1: h − 1 ∼ π t , a h ∼ U ( A ) , ¯ a h +1 ∼ U
( T A ∪ ¯
T A ) .
Calculation of PO-bilinear rank Finally, we prove a PSR belongs to the PO-bilinear class.
Lemma 15 (PO-bilinear decomposition) .
Let Q be a minimum core test set contained in T .
The PSR model has PO-bilinear rank at most |O| M |A| M |Q| , i.e., there exists two |O| M |A| M |Q| -dimensional mappings W h :
Π × G → R |O| M |A| M |Q| and X h : Π → R |O| M |A| M |Q| such that for any tripe ( π, g ; π ′ ) , we have:
[Equation]
Proof.
We first take expectation conditional on τ ah .
Then, we have
[Equation]
where J π 1 and J π 2 are some two matrices as defined in the proof of Lemma 13 from where we have already known that the π -induced Bellman backup on a value function which has a bilinear form gives back a bilinear form value function.
Rearrange terms, we get:
[Equation]
Now recall that the minimum core test set is Q ⊂ T .
The final step is to argue that q τ lives in a subspace whose dimension is |Q| .
Since Q is a core test set, by definition, we can express q τ ah using [ P ( t | τ ah )] t ∈Q , i.e.,
[Equation]
where the row of K indexed by t ∈ T is equal to k t , where k t is the vector that is used to predict P ( t | τ ah )
=
k ⊤ t [ P ( t | τ ah )]
t ∈Q whose existences is ensured by the definition of PSRs.
This implies that
[Equation]
Finally, we take expectation with respect to τ ah then we get Br h ( π, g ; π ′ ) =
⟨ X h ( π ′ ) , W h ( π, g ) ⟩ such that
[Equation]
The key observation here is that the bilinear rank scales with |Q| but not |T | .
This is good news since we often cannot identify exact minimal core test sets; however, it is easy to find core tests including minimal core tests.
Thus, even if we do not know the linear dimension of a dynamical system a priori, the resulting bilinear rank is the linear dimension of dynamical systems as long as core sets are large enough so that they include minimal core tests.
This will result in the benefit of sample complexity as we will see Section N.
F Generalization of PO-Bilinear AC Class
We extend our previous definition of PO-Bilinear AC framework.
We first present an even more general framework that captures all the previous examples that we have discussed so far.
We then provide two more examples that can be covered by this framework: (1) M -step decodable POMDPs, and (2) observable POMDPs with low-rank latent transition.
Using the result in (2), we can obtain refined results in the tabular setting compared to the result from Section 5.3.2.
The following is a general PO-Bilinear AC Class.
Recall M ( h ) :
= max( h − M, 1) .
We consider one-step future, i.e., K = 1 , but the extension to K > 1 is straightforward.
Comparing to Definition 3, we introduce another class of functions termed as discriminators F and the loss function l .
Definition 9 (General PO-Bilinear AC Class) .
Consider a tuple ⟨ Π , G , l, Π e , F⟩ consisting of a policy class Π , a function class G , a loss function l = { l h } Hh =1 where l h ( · ; f, π, g ) :
H h − 1 × O ×
A × R × O → R , a set of estimation policies Π e := { π e ( π ) : π ∈ Π }
where π eh ( π ) : ¯ Z h → ∆( A ) , and a discriminator class F = {F h } with F h ⊂
[ H h → R ] .
Consider a non-decreasing function ζ : R + → R with ζ (0) = 0 .
The model is a PO-bilinear class of rank d if G is realizable, and there exist W h : Π × G → R d and
X h : Π → R d such that for all π, π ′ ∈ Π , g ∈ G and h ∈
[ H ] ,
[Equation] [Equation]
[Equation] [Equation]
[Equation]
(In M -step decodable POMDPs and POMDPs with low-rank latent transition, we set π e ( π ) =
U ( A ) and in the previous sections, we set π e ( π ′ )
=
π ′ . )
(c
[Equation]
for any π ∈ Π and the corresponding future-dependent value function g π in G .
The first condition states the average Bellman error under π is upper-bounded by the quantity in the bilinear form.
The second condition states that we have a known loss function l that can be used to estimate an upper bound (up to a non-decreasing transformation ζ ) of the value of the bilinear form.
Our algorithm will use the surrogate loss l ( · ) .
As we will show, just being able to estimate an upper bound of the value of the bilinear form suffices for deriving a PAC algorithm.
The discriminator F and the non-decreasing functions ζ give us additional freedom to design the loss function.
For simple examples such as tabular POMDPs and LQG, as we already see, we simply set the discriminator class F = ∅ (i.e., we do not use discriminators) and ζ being the identity mapping.
With this definition, we slightly modify P ROVABLE to incorporate the discriminator to construct constraints.
The algorithm is summarized in Algorithm 3 that is named as D IS P ROVABLE .
There are two modifications: (1) when we collect data, we switch from the roll-in policy π t to the policy π e at time step M ( h ) ; (2) the Bellman error constraint σ th is defined using the loss l together with the discriminator class F h .
The following theorem shows the sample complexity of Algorithm 3.
For simplicity, we direct consider the case where Π , G , F are all discrete.
Assumption 3 (Uniform Convergence) .
Fix h ∈
[ H ] .
Let D ′ h be a set of m i.i.d tuples by executing a 1: M ( h ) − 1 ∼ π t , a M ( h ): h ∼ π e With probability 1 − δ , sup
[Equation]
For h = 1 , we also require
[Equation]
Theorem 8 (Sample complexity of Algorithm 3) .
Suppose we have a PO-bilinear AC class with rank d in Definition 9.
Suppose Assumption 3, sup π ∈ Π
∥ X h ( π ) ∥
≤ B X and sup π ∈ Π ,g ∈G ∥ W h ( π, g )
∥ ≤
[Equation]
where D th is the empirical approximation by executing a 1: M ( h ) − 1 ∼ π t , a M ( h ): h ∼ π e ( π t ) and collecting m i.i.d tuples.
Select policy optimistically as follows
[Equation] [Equation]
[Equation]
By setting T = 2 Hd ln (cid:16) 4 Hd (cid:16) B 2 X B 2 W ζ 2 (˜ ϵ gen ) + 1 (cid:17)(cid:17) , R = ϵ gen where ϵ gen := max
h ϵ gen,h ( m, Π , G , F , δ/ ( TH + 1)) ,
˜ ϵ gen := max h ϵ gen,h ( m, Π , G , F , δ/H ) .
With probability at least 1 − δ , letting π ⋆ = argmax π ∈ Π J ( π ⋆ ) , we have
J ( π ⋆ ) − J (ˆ π ) ≤ H 1 / 2 (cid:2) 4 ζ ( ϵ gen )
+
Tζ (2 ϵ gen ) 2 Hd ln(4 Hd ( B 2 X B 2 W /ζ 2 (˜ ϵ gen ) + 1)) (cid:3) 1 / 2 + 2 ϵ ini .
The total number of samples used in the algorithm is mTH .
This reduces to Theorem 1 when we set ζ as an identify function and π e ( π ′ )
=
π ′ .
When ζ − 1 ( · ) is a strongly convex function, we can gain more refined rate results.
For example, when ζ ( x ) = √ x , i.e., ζ − 1 ( x ) = x 2 , with ϵ gen = O (1 / √ m ) , the above theorem implies a slow sample complexity rate 1 /ϵ 4 .
However, by leverage the strong convexity of the square function ζ − 1 ( x ) := x 2 , a refined analysis can give the fast rate 1 /ϵ 2 .
We will see such two examples in the next sections.
G Examples for Generalized PO-Bilinear AC Class
We demonstrate that our generalized framework captures two models: (1) M -step decodable
POMDPs, and (2) observable POMDPs with the latent low-rank transition.
In this section, we assume r h ∈ [0 , 1] for any h ∈
[ H ] .
G.1 M -step decodable POMDPs
The example we include here is a model that involves nonlinear function approximation but has a unique assumption on the exact identifiability of the latent states.
Example 5 ( M -step decodable POMDPs [19]) .
There exists an unknown decoder ι h : ¯ Z h → S , such that for every reachable trajectory ( s 1: h , a 1: h − 1 , o 1: h ) , we have s h =
ι h (¯ z h ) for all h ∈
[ H ] .
Note that when M = 0 , this model is reduced to the well-known Block MDP model [17, 51, 78].
Existence of future-dependent value functions.
From the definition, using a value function
V πh ( z h − 1 , s h ) over z h −
1 ∈
Z h − 1 , s h ∈ S , we can define a future-dependent value function v πh :
Z h − 1 × O → R as
[Equation]
since it satisfies
[Equation]
This is summarized in the following lemma.
Lemma 16 (Existence of future-dependent value functions in M -step decodable POMDPs) .
In
M -step decodable POMDPs, future-dependent value functions exist.
M -step decodable POMDPs showcase the generality of future-dependent value functions , which not only capture standard observability conditions where future observations and actions are used to replace belief states (e.g., observable tabular POMDPs and observable LQG), but also capture a model where history is used to replace latent states.
PO-Bilinear Rank.
Next, we calculate the PO-bilinear rank based on Definition 9.
In the tabular case, we can naïvely obtain the PO-bilinear decomposition with rank |O| M |A| M |S| following
Example 1.
Here, we consider the nontabular case where function approximation is used and |O| can be extremely large.
We define the following Bellman operator associated with π at step h :
[Equation]
Note that above we use the ground truth decoder ι h to decode from ¯ z h to its associated latent state s h .
The existence of this Bellman operator B πh is crucially dependent on the existence of such decoder ι h .
We show that M -step decodable POMDPs satisfy the definition in Definition 9.
We assume that the latent state-wise transition model is low-rank.
In MDPs, this assumption is widely used in [76, 36, 2, 71].
Here, we do not need to know µ, ϕ in the algorithm.
Assumption 4 (Low-rankness of latent transition) .
Suppose T is low-rank, i.e., T ( s ′ | s, a ) =
⟨ ϕ ( s, a ) , µ ( s ′ )
⟩ ( ∀ ( s, a, s ′ )) where ϕ, µ are (unknown) d -dimensional features.
As technical conditions, we suppose ∥ ϕ ( s, a ) ∥ ≤ 1 for any ( s, a ) and | (cid:82) µ ( s ) v ( s ) d ( s ) | ≤ √ d for any ∥ v ∥ ∞ ≤ 1 .
Lemma 17 (Bilinear decomposition of low-rank M -step decodable POMDPs ) .
Suppose Assumption 4, ∥G h ∥ ∞ ≤ H, ∥F h ∥ ∞ ≤ H , r h ∈ [0 , 1] for any h ∈
[ H ] .
Assume a discriminator class is Bellman complete, i.e.,
[Equation]
for any h ∈ [ H ] .
The loss function is designed as
[Equation]
Then, there exist W h
( π, g ) , X h ( π ′ )
so that the PO-bilinear rank is at most d such that
[Equation] [Equation]
and
[Equation]
Proof.
The proof is deferred to Section Q.2.
Note that (12), (13), (14) correspond to (a), (b), (c) in
Definition 9.
We use the most general bilinear class definition from Definition 9, where ζ ( a ) = |A| M/ 2 a 1 / 2 for scalar a ∈ R
+ .
Hence ζ is a non-decreasing function ( ζ is non-decreasing in R + ).
The proof of the above lemma leverages the novel trick of the so-called moment matching policy introduced by [19].
When the latent state and action space are discrete, it states that the bilinear rank is |S||A| , which is much smaller than |O| M |A| M |S| .
Note we here introduce − 0 .
5 f
(¯ z h ) 2 in the loss function (11) to induce strong convexity w.r.t f as in [70, 14, 10], which is important to obtain the fast rate later.
The concrete sample complexity of P ROVABLE (Algorithm 3) for this model is summarized in the following.
Recall that the bilinear rank is d where d is the rank of the transition matrix.
We set
G h ⊂ [ ¯ Z h → [0 , H ]] .
Then, we have the following result.
Theorem 9 (Sample complexity for M -step decodable POMDPs (Informal)) .
Suppose Assumption 4, Bellman completeness, ∥G h ∥ ∞ ≤ H, ∥F h ∥ ∞ ≤ H , r h ∈ [0 , 1] for any h ∈
[ H ] .
With probability 1 − δ , we can achieve J ( π ⋆ ) − J (ˆ π ) ≤ ϵ when we use samples at most
[Equation]
Here, polylog( d, H, |A| , 1 /ϵ, ln( | Π max | ) , ln( |F max | ) , ln( |G max | ) , ln(1 /δ )) are omitted.
The followings are several implications.
First, the error rate scales with O (1 /ϵ 2 ) .
As we promised, by leveraging the strong convexity of loss functions, we obtain a rate
O (1 /ϵ 2 ) , which is faster than
O (1 /ϵ 4 ) that are attained when we naively invoke Theorem 8 with ξ ( x ) ∝ √ x .
Secondly, the error bound incurs |A| M .
As showed in [19], this is inevitable in M -step decodable POMDPs.
Thirdly, in the tabular case, when we use the naïve function classes for G , F , Π , i.e., G h = { ¯ Z h → [0 , H ] } ,
F h = { ¯ Z h → [0 , H ] } , Π h
= { ¯ Z h → ∆( A ) } , the bound could incur additional |O| M since the complexity of the function classes can scale with respect to ( |O||A| )
M (e.g., log( |G h | ) can be in the order of O ( |O| M |A M ) , and similarly for log( |F h | ) , ln(Π h ) ).
However, when we start form a realizable model class that captures the ground truth transition and omission distribution, we can remove |O| M .
See Section G.2.4 for an example.
Note that [19] uses a different function class setup where they assume one has an M memory-action dependent Q function class Q h : ¯ Z h ×A → R which contains Q ⋆h (¯ z h , a ) while we use the actor-critic framework v πh , π .
The two function class setups are not directly comparable.
Generally, we mention that such optimal Q ⋆ with truncated history does not exist when the exact decodability does not hold (e.g., such Q ⋆ with truncated history does not exist in LQG).
This displays the potential generality of the actor-critic framework we propose here.
G.2 Observable POMDPs with Latent Low-rank Transition: a model-based perspective
The final example we include in this work is a POMDP with the latent low-rank transition.
We first introduce the model, and then we introduce our function approximation setup and show the sample complexity.
Finally, we revisit the sample complexity for observable tabular POMDPs and
M -step decodable tabular POMDPs using the improved algorithm that elaborates on the model-based approach in this section.
Example 6 (Observable POMDPs with latent low-rank transition) .
The latent transition T ( s ′ | s, a ) is factorized as T ( s ′ | s, a )
= µ ⋆ ( s ′ )
⊤ ϕ ⋆ ( s, a ) , ∀ s, a, s ′
where µ ⋆ : S → R d and ϕ
⋆ : S ×
A → R d .
The observation |O| × |S| matrix O has full-column rank.
In the tabular POMDP example, we have d ≤ |S| .
However in general d can be much smaller than |S| .
Note that in this section, we will focus on the setting where S , O are discrete to avoid using measure theory languages, but their size could be extremely large.
Particularly, our sample complexity will not have explicit polynomial or logarithmic dependence on |O| , |S| , instead it will only scale polynomially with respect to the complexity of the hypothesis class and the rank d .
Model-based function approximation.
Our function approximation class consists of a set of models M = { ( µ, ϕ, O ) } where µ, ϕ together models latent transition as µ ( · ) ⊤ ϕ ( s, a ) ∈ ∆( S ) , and O : S → ∆( O ) models O , and O is full column rank.
For notation simplicity, we often use θ :=
( µ, ϕ, O ) ∈ M to denote a model ( µ, ϕ, O ) .
We impose the following assumption.
Assumption 5 (Realizability) .
We assume realizability, i.e., ( µ ⋆ , ϕ
⋆ , O ) ∈ M .
We assume M is discrete, but |M| can be large such that a linear dependence on |M| in the sample complexity is not acceptable.
Our goal is to get a bound that scales polynomially with respect to ln( |M| ) , which is the standard statistical complexity of the discrete hypothesis class M .
Next, we construct Π , G , F using the model class M .
Given θ := ( µ, ϕ, O ) , we denote π θ as the optimal M -memory policy, i.e., the M -memory policy that maximizes the total expected reward.
We set
[Equation]
We consider the value function class for θ :=
( µ, ϕ, O ) with O being full column rank.
For each θ , we can define the corresponding value function of the policy π at h ∈
[ H ] : V πθ ; h ( z h − 1 , s h ) :
Z h − 1 × S → R .
Then, since O is full column rank, as we see in the proof of Lemma 1, a corresponding future-dependent value function is
[Equation]
⟩ where V πθ ; h ( z h − 1 , s h ) =
⟨ f πθ ; h , 1 ( z ) ⊗ 1 ( s ) ⟩ .
Then, we construct G = {G h } as:
[Equation]
By construction, since θ ⋆ :=
( µ ⋆ , ϕ ⋆ , O ) ∈ M , we must have g π ∈ G , ∀ π ∈ Π , which implies G is realizable (note g πh =
g πθ ⋆ ; h ).
Here, from the construction and the assumption r h ∈ [0 , 1] for any h ∈ [ H ] , we have |G h | ≤ |M| 2 and ∥G h ∥ ∞ ≤ H/σ 1 , which can be seen from ∀ ( z, o ); ⟨ f πθ ; h , 1 ( z ) ⊗ O † 1 ( o ) ⟩ ≤ ∥ f πθ ; h ∥ ∞
∥ 1 ( z ) ⊗ O † 1 ( o ) ∥ 1 ≤ H ×
∥
O † 1 ( o ) ∥ 1 ≤ H/σby assuming ∥ O † ∥ 1 ≤ 1 /σ 1 and ∥ f πθ ; h ∥ ∞ ≤ H .
To construct a discriminator class F , we first define the Bellman operator B πθ ; h for π ∈ Π ,
h ∈
[ H ] , θ ∈ M :
[Equation]
where H h is the whole history space up to h ( τ h
= ( a 1: h − 1 , o 1: h ) , and ¯ z h is just part of this history) and P θ ( o h +1 |
τ h , a h ) is the probability of generating o h +1 conditioned on τ h , a h under model θ .
Then, we construct F = {F h } such that
[Equation]
so that we can ensure the Bellman completeness:
[Equation]
noting B πθ ⋆ ; h = B πh .
Here, from the construction, |F h | ≤ |M| 2 × |M| 2 × |M| 2 = |M| 6 and ∥F h ∥ ∞ ≤ 3 H/σ 1 .
We define the loss as the same as the one we used in M -step decodable POMDPs, except that our discriminators now take the entire history as input:
[Equation]
Finally, as in the case of M -step decodable POMDPs (Lemma 17), we get the following lemma that states that our model is a PO-bilinear AC class (Definition 9) under the following model assumption.
Assumption 6.
We assume ∥
O † ∥ 1 ≤ 1 /σ 1 for any O in the model.
Suppose µ ( · ) ⊤ ϕ
( s, a ) ∈ ∆( S ) for any ( s, a ) , µ ( · ) and ϕ ( · ) in the model.
Suppose ∥ ϕ
( s, a ) ∥ ≤ 1 for any ϕ in the model and ( s, a ) ∈
S×A .
Suppose for any v : S → [0 , 1] and for any µ in the model, we have ∥ (cid:82) v ( s ) µ ( s )d( s ) ∥ 2 ≤ √ d .
Lemma 18 (PO-bilinear decomposition for Observable POMDPs with low-rank transition) .
Suppose
Assumption 5, 6.
Consider observable POMDPs with latent low-rank transition.
Set G as in (15) ,
F as in (16) and l as in (17) .
Then, there exist W h
( π, g ) , X h ( π ′ )
that admits the PO-bilinear rank decomposition in Definition 9 with rank d .
The above lemma ensures that the PO-bilinear rank only depends on d , and is independent of the length of the memory.
For example, in the tabular case, it is |S| .
Next, we show the output from D IS P ROVABLE can search for the best in class M -memory policy as follows.
Theorem 10 (Sample complexity of D IS P ROVABLE for observable POMDPs with latent low-rank transition) .
Consider observable POMDPs with latent low-rank transition.
Suppose Assumption 5, 6.
With probability 1 − δ , we can achieve J ( π ⋆ ) − J (ˆ π ) ≤ ϵ when we use samples at most
[Equation]
Here, we omit polylog( d, H, |A| , ln(1 /δ ) , ln( |M| ) , 1 /σ 1 , 1 /ϵ ) .
Here, we emphasize that there is no explicit polynomial or logarithmic dependence on |S| and |O| , which permits learning for large state and observation spaces.
We also do not have any explicit polynomial dependence on |O| M , as we construct Π and G from the model class M which ensures the complexities of π and G are in the same order as that of M .
G.2.1 Global Optimality
We show a quasi-polynomial sample complexity bound for competing against the globally optimal policy π ⋆ gl .
To compete against the globally optimal policy π ⋆ gl , we need to set M properly.
We use the following lemma.
The proof is given in Section R.
Lemma 19 (Near global optimaltiy of M -memoruy policy) .
Consider ϵ ∈ (0 , H ] , and a POMDP with low-rank latent transition and O being full column rank with ∥ O † ∥ 1 ≤ 1 /σ 1 .
When M = Θ( C (1 /σ 1 ) − 4 ln( dH/ϵ )) (with C being some absolute constant), there must exists an M -memory policy π ⋆ , such that J ( π ⋆ gl ) − J ( π ⋆ ) ≤
ϵ
Note that the memory M above is independent of |S| instead it only depends on the rank d .
To prove the above lemma, we first show a new result on belief contraction for low-rank POMDPs under the ℓ 1 -based observability.
The proof of the belief contraction borrows some key lemma from [23] but extends the original result for small-size tabular POMDPs to low-rank POMDPs.
We leverage the linear structure of the problem and the G-optimal design to construct an initial distribution over S that can be used as a starting point for belief propagation along the memory.
We conclude the study on the POMDPs with low-rank latent transition by the following theorem, which demonstrates a quasi-polynomial sample complexity for learning the globally optimal policy.
Theorem 11 (Sample complexity of D IS P ROVABLE for POMDPs with low-rank latent transition — competing against π ⋆ gl ) .
Consider observable POMDPs with latent low-rank transition.
Fix some ϵ ∈ (0 , H ) , δ ∈ (0 , 1) .
Suppose Assumption 5, 6.
We construct Π , G , F , and the loss l as we described above.
With probability at least 1 − δ , when M = Θ( Cσ − 4
ln( dH/ϵ )) , D IS P ROVABLE outputs a ˆ π such that J ( π ⋆ gl ) − J
(ˆ π ) ≤ ϵ , with number of samples scaling
[Equation]
Remark 6 (Comparison to [73]) .
We compare our results to the very recent work [73] that studies
POMDPs with the low-rank latent transition.
The results are in general not directly comparable, but we state several key differences here.
First, [73] considers a special instance of low-rank transition, i.e., [73] assumes T has low non-negative rank, which could be exponentially larger than the usual rank [2].
Second, [73] additionally assumes short past sufficiency , a condition which intuitively says that for any roll-in policy, the sufficient statistics of a short memory is enough to recover the belief over the latent states, and their sample complexity has an exponential dependence on the length of the memory.
While our result also relies on the fact that the globally optimal policy can be approximated by an M -memory policy with small M , this fact is derived directly from the standard observability condition.
G.2.2 Revisiting Observable Undercomplete Tabular POMDPs
We reconsider the sample complexity of undercomplete tabular POMDPs using Theorem 10.
In this case, we will start from a model class that captures the ground truth latent transition T and omission distribution O .
By constructing ϵ -nets over the model class,we can set ln( |M| ) =
˜ O ( |S| 3 |O||A| ) since T , O have |S| 2 |A| and |O||S| many parameters, respectively.
Besides, the PO-bilinear rank is d = |S| .
Therefore, the sample complexity is
[Equation]
We leave the formal analysis to future works.
Compared to results in Section 5.3.2, there is no |O| M term.
This is due to two improvements.
The first improvement is that we refine the rank from |O| M |A| M |S| to |S| .
The second improvement is we model the future-dependent value function class and policy class starting from the model class whose complexity has nothing to do with the length of memory M (note that previously, from a pure model-free perspective, the statistical complexity of G can scale as |O| M |A| M |S| in the worst case).
G.2.3 Revisiting Observable Overcomplete POMDPs
We reconsider the sample complexity of overcomplete tabular POMDPs using Theorem 10 with slight modification to incorporate multi-step future.
Suppose ∥{ O K } † ∥ 1 ≤ 1 /σ 1 (recall O K is defined in
Lemma 5 in Section C.2).
Then, we can achieve a sample complexity
[Equation]
since the PO-bilinear rank is |S| .
Note that there is no |O| M + K dependence, since both the policy class and the future-dependent value function class are built from the model class whose complexity has nothing to do with M, K .
Note that due to our definition of O K , there is no |A| K term.
However, when we use a different definition, for instance, min a ′ h :
h
+ K
[Equation] [Equation]
1 ∥{ O K ( a ′ h : h + K − 2 ) } † ∥ 1 ≤ 1 /α 1 (recall O K ( a ′ h :
h
+ K − 2 ) is defined in Section C.2), we would incur |A| K .
This is because if we only know that there is an unknown sequence of actions a ′ h :
h + K − 2 such that O K ( a ′ h : h + K − 2 ) is full column rank, we need to use uniform samples |A| K in the importance sampling step to identify such a sequence.
More formally, we can see that
[Equation]
G.2.4 Revisiting M -step Decodable Tabular POMDPs
We reconsider the sample complexity of tabular M -step decodable POMDPs by constructing F , G , Π from the model class M as we did for the low-rank POMDP.
In this case, by constructing ϵ -nets, we can set ln( |M| ) =
˜ O ( |S| 3 |O||A| ) since T , O have |S| 2 |A| and |O||S| parameters, respectively.
Therefore, the sample complexity is
[Equation]
Again, we leave the formal analysis to future works.
Compared to the naive result mentioned after
Theorem 9 where ln( G ) , ln(Π) could scale in the order of |O| M in the tabular case, we do not have |O| M dependence here.
H Proof of Theorem
We fix the parameters as in Theorem 1.
Let
[Equation]
We define
[Equation]
Then, by our assumption 2 with probability 1 − δ , we ∀ t ∈ [ T ] , ∀ h ∈
[ H ]
[Equation]
Hereafter, we condition on the above events.
We first show the following lemma.
Recall
[Equation]
Lemma 20 (Optimism) .
Set R :=
ϵ 2 gen .
For all t ∈ [ T ] , ( π ⋆ , g π ⋆ ) is a feasible solution of the constrained program.
Furthermore, we have J ( π ⋆ ) ≤ E [ g t 1 ( o 1 )]
+ 2 ϵ ini for any t ∈
[ T ] , where g t is the future-dependent value function selected by the algorithm in iteration t .
Proof.
For any π , we have
[Equation]
since g π is a future-dependent value function in G .
This is because
[Equation]
Thus,
[Equation]
using (19) noting π ⋆ ∈ Π
[Equation]
g π ⋆ ∈ G .
This implies
[Equation]
Hence, ( π ⋆ , g π ⋆ ) is a feasible set for any t ∈ [ T ] .
Then, we have
[Equation]
Using the construction of algorithm)
[Equation]
Remark 7.
Note that
[Equation]
holds for general future-dependent value functions g π inDefinition 6 .
Thus, the statement goes through even if we use Definition 6.
Next, we prove the following lemma to upper bound the per step regret.
Lemma 21.
For any t ∈ [ T ] , we have
J ( π ⋆ ) − J (ˆ π ) ≤
H (cid:88) h = |⟨
W h ( π t , g t ) , X h ( π t ) ⟩| + 2 ϵ ini .
Proof.
J ( π ⋆ ) − J (ˆ π ) ≤ 2 ϵ ini + E [ g t 1 ( o 1 )] − J ( π t ) (From optimism) =
2 ϵ ini
+
H (cid:88) h
=
E [ g th (¯ z h ) − { r h + g th +
(¯ z h +1 ) } ; a 1: h ∼ π t ] (Performance difference lemma) ≤ 2 ϵ ini
+
H (cid:88) h
= | E [ g th (¯ z h ) − { r h
+ g th +
(¯ z h +1 ) } ; a 1: h ∼ π t ] | = 2 ϵ ini
+
H (cid:88) h = |⟨
W h ( π t , g t ) , X h ( π t ) ⟩| .
(First assumption in Definition 3)
Lemma 22.
Let Σ t,h = λI + (cid:80) t − 1 τ =
X h ( π τ ) X h ( π τ ) ⊤ .
We have
T
[Equation]
h
[Equation]
X
[Equation]
(cid:115)
[Equation]
(cid:18)
[Equation]
dλ (cid:19) .
Proof.
We fix h ∈
[ H ] .
Here, we have Σ t,h = λI + (cid:80) t − 1 τ =
X h ( π τ ) X h ( π τ ) ⊤ .
From the elliptical potential lemma in [1, Lemma G.2], we have
[Equation]
Then,
[Equation]
Lemma 23.
[Equation]
Proof.
We have
[Equation]
The first term is upper-bounded by λB 2 W .
The second term is upper-bounded by t − 1 (cid:88) τ = ⟨
[Equation]
( π t , g t ) , X h ( π τ )
⟩= t − 1 (cid:88) τ =
[Equation]
E [ l
[Equation]
¯ z
[Equation]
a h
[Equation]
r
[Equation]
o h
+
[Equation]
π t
[Equation]
g t )
[Equation]
a 1: h
[Equation]
(First assumption in Definition 3)
[Equation]
From the first line to the second line, we use the definition of bilinear rank models.
From the second line to the third line, we use ( a
+ b ) 2 ≤ 2 a
+ 2 b 2 .
In the last line, we use the constraint on ( π t , g t ) .
Combining lemmas so far
[Equation]
we have
[Equation] [Equation]
(Use Lemma 22 and Lemma 23)
We set λ such that B 2 X /λ =
B 2 W B 2 X /ϵ 2 gen + 1 and T = (cid:6) 2 Hd ln(4 Hd ( B 2 X B 2 W / ˜ ϵ gen + 1))
(cid:7) .
Then,
[Equation]
(cid:19)(cid:19)
[Equation]
since a ln( bT ) /T ≤ 1 when T = 2 a ln(2 ab ) .
Finally, the following holds
[Equation]
I Sample Complexity for Finite Function Classes
Consider cases where Π and G are finite and the PO-bilinear rank assumption is satisfied.
When Π and G are infinite hypothesis classes, |F| and |G| are replaced with their L ∞ -covering numbers, respectively.
Theorem 12 (Sample complexity for discrete Π and G ) .
Let ∥G h ∥ ∞ ≤ C G , r h ∈ [0 , 1] for any h ∈
[ H ] and the PO-bilinear rank assumption holds with PO-bilinear rank d .
By letting | Π max | = max h | Π h | , |G max | = max h |G h | , with probability 1 − δ , we can achieve J ( π ⋆ ) − J (ˆ π ) ≤ ϵ when we use samples at most
[Equation]
Here, polylog
[Equation]
(1 /δ ) , ln( B X ) , ln( B W ) , ln(1 /δ ) , (1 /ϵ )) are omitted.
Proof.
We derive the above result.
First, we check the uniform convergence result.
Then,
[Equation]
Thus
[Equation]
we need to set m such that
[Equation]
where c is some constant and
[Equation]
By organizing the term
[Equation]
the following m is sufficient
[Equation]
Using Lemma
[Equation]
the following m satisfies the condition:
[Equation]
Combining all together, the sample complexity is mTH , i.e.,
[Equation]
J Sample Complexity in Observable HSE POMDPs
We revisit the existence of future-dependent value functions by taking the norm constraint into account.
Then, we consider the PO-bilinear decomposition with certain B X ∈ R and B W ∈ R .
Next, we calculate the uniform convergence result.
Finally, we show the sample complexity result.
We use the following assumptions
Assumption 7.
For any h ∈ [ H ] , the following holds:
V πh ( z h − 1 , s ) =
⟨ θ πh , ϕ h ( z h − 1 , s ) ⟩ . 2.
There exists a matrix K h such that E o ∼ O ( s )
[ ψ h ( z h − 1 , o )]
=
K h ϕ h ( z h − 1 , s ) (i.e., conditional embedding of the omission distribution), 3.
∥
ϕ h ( · ) ∥ ≤ 1 , ∥ ψ h ( · ) ∥ ≤ 1 , ∥ θ πh ∥ ≤ Θ V , 0 ≤ r h ≤ 1 , 4.
There exists a matrix T π ; h such that E [ ϕ h ( z h , s h +1 )
| z h − 1 , s h ; a h ∼ π ]
= T π ; h ϕ h ( z h − 1 , s h )
(i
[Equation]
e., conditional embedding of the transition) 5.
Π is finite.
We define
[Equation]
Existence of future-dependent value functions.
We show future-dependent value functions exist.
This is proved by noting
E
[Equation]
Thus, ⟨ ( K † h ) ⊤ θ πh , ψ h (¯ z h ) ⟩ is a future-dependent value function.
The radius of the parameter space is upper-bounded by Θ V /σ min ( K ) .
Hence, we set
[Equation]
K ) } .
Then, the realizability holds.
PO-bilinear decomposition.
Recall we derive the PO-bilinear decomposition in Section C.4.
Consider a triple ( π ′ , π, g ) with g h ( · )
=
θ ⊤ h ψ h ( · ) and g πh = ⟨ θ ⋆h , ψ h ( · ) ⟩ , we have:
[Equation]
which verifies the PO-bilinear structure, i.e.,
[Equation]
and shows that the PO-bilinear rank is at most d ϕ = max
h d ϕ h .
Thus, based on the above PObilinear decomposition, we set ∥ B X ∥ = 1 , ∥ B W ∥ =
2(1 + σ max ( T ))
σ max ( K )Θ V /σ min ( K ) .
This is because
[Equation]
and ∥
[Equation]
In the above, we use Jensen’s inequality.
Uniform convergence.
To invoke Theorem 1, we show the uniform convergence result.
Lemma 24 (Uniform convergence of loss functions) .
Let C
=
Θ V / ( σ min ( K )) .
Then, with probability 1 − δ ,
[Equation]
m and
[Equation]
m .
Proof.
Let C
=
Θ V /σ min ( K ) .
Define N ϵ,h as an ϵ -net for G h .
Then, |N ϵ,h | ≤ (1 + C/ϵ ) d .
Then,
[Equation]
Besides, for fixed π ∈ Π , θ h ∈
N ϵ,h , θ h +1 ∈ N ϵ,h
+1 , we have
[Equation]
Then, for ∀ π ∈ Π , ∀ θ h ∈ N ϵ,h , ∀ θ h +1 ∈ N ϵ,h
+1 , we have |{ E D − E } [
[Equation]
A
[Equation] [Equation]
Hence, for any g h = ⟨ θ h , ψ h ⟩ ∈ G h , g h
+1 =
⟨ θ h +1 , ψ h +1 ⟩ ∈
G h
+1 ,
[Equation]
N ϵ
[Equation]
N
[Equation]
By taking ϵ = 1 /m , we have ∀ π ∈ Π , ∀ g h ∈
G h , ∀ g
h +1 ∈ G h
+1 :
[Equation]
Similarly,
[Equation]
m .
Finally
[Equation]
we obtain the PAC bound, we need to find m such that
[Equation]
where c is some constant and
T = cHd
[Equation] [Equation]
X B 2 W m +
[Equation]
By organizing the term, the following m is sufficient:
[Equation]
By using Lemma 44, we can set
[Equation]
Thus, the final sample complexity is
[Equation]
where
C =
Θ V /σ min ( K ) .
K Sample Complexity in Observable Undercomplete Tabular POMDPs
We revisit the existence of future-dependent value functions.
Then, we show the PO-bilinear rank decomposition.
After showing the uniform convergence lemma, we calculate the sample complexity.
Existence of future-dependent value functions.
In the tabular case, by setting
[Equation]
where 1 ( z ) is a one-hot encoding vector over Z h − 1 , we can regard the tabular model as an HSEPOMDP.
Here is our assumption.
Assumption 8.
(a) 0 ≤ r h ≤ 1 , (b) O is full-column rank and ∥ O † ∥ 1 ≤ 1 /σ 1 for any h ∈
[ H ] .
Note we use the 1 -norm since this choice is more amenable in the tabular setting.
However, even if the norm bound is given in terms of 2 -norm, we can still ensure the PAC guarantee (this is because ∥ O † ∥ 1 / (cid:112) |S| ≤ ∥ O † ∥ 2 ≤ ∥
O † ∥ 1 (cid:112) |O| ).
Here, since we assume the reward lies in [0 , 1] , value functions on the latent state belong to {⟨ θ, ϕ h ( · ) ⟩ : ∥ θ ∥ ∞ ≤ H } .
Here, letting V πh = ⟨ θ πh , ϕ h ⟩ , future-dependent value functions exist by taking ⟨ θ πh , 1 ( z ) ×
O † 1 ( o ) ⟩ .
Hence, we take
[Equation]
so that the realizability holds.
Importantly, we can ensure ∥G h ∥ ∞ ≤ H/σ 1 since
[Equation]
∥ 1 ≤ H/σfor any ( z, o ) ∈
Z h − 1 × O .
Note G h is contained in
[Equation]
This is because each ⟨ θ, 1 ( z ) ⊗ O † 1 ( o ) ⟩ is equal to ⟨ θ ′ , 1 ( z ) ⊗ 1 ( o ) ⟩ for some vector θ ′ ∈ R |Z h − 1 |×|O| .
Here, denoting the component of θ corresponding to z ∈ Z h − 1 by θ z ∈ R |O| , θ ′ is a vector stacking O † θ z for each z ∈ Z h − 1 .
Then, we have
[Equation]
Hence, ∥ θ ′ ∥ 2 ≤ |O| M |A| M × H |O| /σ 1 .
PO-Bilinear decomposition.
Next, recall we derive the PO-bilinear decomposition:
[Equation]
Then, B X = 1 and B W = 4 H |O| M +1 |A| M /σ 1 .
We use ∥ K ⊤ h ∥ 2 = ∥
O h ∥ 2 ≤ 1 , ∥ T ⊤ π : h ∥ 2 ≤ 1 .
This is because
[Equation]
In the last line, we use (21).
Uniform convergence.
Then, we can obtain the following uniform convergence lemma.
Lemma 25.
Let C
= H/σ 1 and d ψ = |O| M +1 |A| M .
Then, with probability 1 − δ ,
[Equation]
m and
[Equation]
Proof.
Let d ϕ = |S||O|
M |A| M , d ψ = |O| M +1 |A| M .
Define N ϵ,h as an ϵ -net for { θ : ∥ θ ∥ 2 ≤ C } with respect to L 2 -norm.
Define N ′ ϵ,h as an ϵ -net for Π h : ¯ Z h → ∆( A ) with respect to the following norm:
[Equation]
Then, |N ϵ,h | ≤ (1 + C/ϵ ) d , |N ′ ϵ,h
| ≤ (1 + 1 /ϵ )
d ψ |A| .
Let g h
= ⟨ θ h , ψ h ⟩ , g ⋄ h
= ⟨ θ ⋄ h , ψ h ⟩ where ψ h is a one-hot encoding vector over ¯ Z h .
Then, when ∥ θ h − θ ⋄ h ∥ 2 ≤ ϵ, ∥ θ h +1 − θ ⋄ h +1 ∥ 2 ≤ ϵ, ∥ π h − π ⋄ h ∥ 1 ≤ ϵ , we have
[Equation]
Besides, for fixed π ∈ N ′ ϵ,h , θ h ∈ N ϵ,h , θ h +1 ∈
N ϵ,h
+1 , we have |{ E D
[Equation] [Equation]
m .
Then, for ∀ π ∈ N ′ ϵ,h , ∀ θ h ∈ N ϵ,h , ∀ θ h +1 ∈ N ϵ,h
+1 , we have |{ E D − E }
[ |A| π h ( a h | ¯ z h ) { g h (¯ z h ; θ h ) − r
h − g h +1 (¯ z h +1 ; θ h +1 ) } ] | ≤ |A| (1 + 2 C ) (cid:115) ln
( |N ′ ϵ,h
||N ϵ,h ||N ϵ,h +1 | /δ )
m .
Hence, for any π h ∈ Π h , g h = ⟨ θ h , ψ h ⟩ ∈
G h , g h +1 =
⟨ θ h +1 , ψ h +1 ⟩ ∈
G h +1 , |{ E D − E } [ |A| π h ( a h |
¯ z h ) {
g h (¯ z h ; θ h ) − r
h − g h +1 (¯ z h +1 ; θ h +1 ) } ] | ≤ |A| (1 + 2 C ) (cid:115) ln
( |N ′ ϵ,h
||N ϵ,h ||N ϵ,h +1 | /δ ) m
+ 3 |A|
Cϵ.
By taking ϵ = 1 /m , we have ∀ π ∈ Π , ∀ g h ∈
G h , ∀ g
h +1 ∈ G h +1 ; |{ E D − E }
[ |A| π h ( a h | ¯ z h ) { g h (¯ z h ) − r
h − g h +1 (¯ z h +1 ) } ] | ≤ |A|{ 1 + 2 C } (cid:114) { 2 d ψ ln(1 + Cm ) + d ψ |A| ln(1 + m ) + ln(1 /δ ) } m + 3 |A| C m ≤ 10 |A| C (cid:114) { d ψ |A| ln(1 + Cm ) + ln(1 /δ ) } m .
Sample Complexity.
Finally, we obtain the PAC bound.
We need to find m such that
[Equation]
where c is some constant and
[Equation]
By organizing terms
[Equation]
we get (cid:114)
[Equation]
Thus
[Equation]
we need to set
[Equation]
Hence, the sample complexity is
[Equation]
By some algebra
[Equation]
it is
[Equation]
Later
[Equation]
we prove we can remove |O| M using the more refined analysis in Section Q.
Global optimality.
We use a result in the proof of [23, Theorem 1.2].
We just set M =
C (1 /σ 1 ) 4 ln( SH/ϵ ) .
Note their assumption 1 is satisfied when ∥
O † ∥ 1 ≤ (1 /σ 1 ) .
More specifically, assumption 1 in [23] requires for any b and b ′ , we have
[Equation]
This is proved as follows.
Note for any e, e
[Equation]
[Equation]
Then, by setting e = O b and e ′ =
O b ′ , the assumption 1 is ensured.
Here, we use O † O = I .
L Sample Complexity in Observable Overcomplete Tabular POMDPs
We first gave an overview of the result.
Then, we move to the detail.
L.1 Summary
We consider obvercomplete tabular POMDPs.
In this case, the PO-bilinear rank is at most |O| M |A| M |S| .
We suppose r h ∈ [0 , 1] for any h ∈ [ H ] .
Assuming O K is full-column rank, to satisfy the realizability, we set G h
= {⟨ θ, 1 ( z ) ⊗ { O K } † 1 ( t K ) ⟩ | ∥ θ ∥ ∞ ≤ H } where ∥ O K ∥ 1 ≤ 1 /σand
( z ) , 1 ( t K ) are one-hot encoding vectors over Z h − 1 and O K × A K − 1 , respectively.
We set Π h
=
[ ¯ Z h → ∆( A )] .
Then, the following holds.
Theorem 13 (Sample complexity for overcomplete tabular models) .
With probability 1 − δ , we can achieve J ( π ⋆ ) − J (ˆ π ) ≤ ϵ when we use samples at most ˜ O (cid:0) |S| 2 |A| 3 M + K +2 |O| 3 M + K H 6 (1 /ϵ ) 2 (1 /σ 1 ) 2 ln(1 /δ )
(cid:1) .
Here, polylog( |S| , |O| , |A| , H, 1 /σ 1 , ln(1 /δ )) are omitted.
When we use K-step futures, in the above theorem, we additionally incur |A| K |O| K , which is coming from a naive parameterization of G h .
In Section G.2.3, we will see that under the model-based learning perspective (i.e., we parameterize T , O first and then construct Π and G using the model class), we will get rid of the dependence |O| M + K and |A| K .
This is because the complexity of the model class is independent of M or K (i.e., number of parameters in T , O are O ( |S| 2 |A||O| ) ).
L.2 Detail
To simplify the presentation, we focus on the case when π out =
U ( A ) .
Existence of future-dependent value functions.
In the tabular case, by setting
[Equation]
where 1 ( z ) is a one-hot encoding vector over Z h − 1 and 1 ( t K ) is a one-hot encoding vector over
Z K =
O K
× A K − 1 , we can regard the tabular model as an HSE-POMDP.
Here is our assumption.
Assumption 9.
(a) 0 ≤ r h ≤ 1 , (b) O K is full-column rank and ∥{ O K } † ∥ 1 ≤ 1 /σ 1 .
Recall we define O K in Lemma 5.
Since we assume the reward lies in [0 , 1] , value functions on the latent state belong to {⟨ θ, ϕ h ( · ) ⟩ : ∥ θ ∥ ∞ ≤ H } .
Here, letting V πh ( · ) = ⟨ θ πh , ϕ h ( · ) ⟩ , future-dependent value functions exist by taking ⟨ θ πh , 1 ( z ) ⊗ { O K } † 1 ( t K ) ⟩ .
Hence, we take
[Equation]
⟨ θ πh , 1 ( z ) ⊗ { O K } † 1 ( t K ) ⟩ ; ∥ θ πh ∥ ∞ ≤ H } so that the realizability holds.
Importantly, we can ensure ∥G h ∥ ∞ ≤ H/σ 1 as in Section K.
Then, as in Section K, G h is contained in
[Equation]
(cid:9) .
PO-bilinear decomposition.
Next, we derive the PO-bilinear decomposition:
E
[Equation] [Equation]
r h
[Equation]
⊤
[Equation] [Equation]
h
[Equation] [Equation]
[Equation] [Equation]
a 1:
[Equation]
− 1 ∼ π ′ ] ⟩ .
Then, B X = 1 and B W = 4 H |O| M +1 |A| M /σ 1 .
We use ∥ K h ∥ 2 = ∥
O K ∥ 2 ≤ 1 , ∥ T ⊤ π : h ∥ 2 ≤ 1 .
Sample Complexity.
We can follow the same procedure in the proof of Section K. Let d ϕ = |S||O|
M |A| M , d ψ = |O| M + K |A| M + K − 1 .
Hence, the sample complexity is
[Equation]
By some algebra, the above is
[Equation]
Using the more refined analysis later, we show we can remove |O| 3 M + K in Section Q.
M Sample Complexity in LQG
In this section, we derive the sample complexity in LQG.
We first explain the setting.
Then, we prove the existence of future-dependent value functions.
Lemma 3 is proved there.
Furthermore, we show the PO-bilinear rank decomposition in LQG.
We prove Lemma 3 there.
Next, we show the uniform convergence result in LQG.
Finally, by invoking Theorem 1, we calculate the sample complexity.
We study a finite-horizon discrete time LQG governed by the following equation:
[Equation]
where ϵ h is Gaussian noise with mean 0 and noise Σ ϵ and τ h is a Gaussian noise with meanand Σ τ .
We use a matrix O instead of C to avoid notational confusion later.
With a linear policy π h ( a h | o h , z
h − 1 ) =
δ ( a h =
U 1 h
o h
+ U 2 h
z h − 1 ) , this induces the following system:
[Equation]
 where z ′ h is the vector removing ( o h , a h ) from z h
and I ′ is a matrix mapping z h to z ′ h .
This is derived by
[Equation]
We suppose the system is always stable in the sense that the operator norm of Ξ 1 h ( π ) is upper-bounded by 1 .
Here is the assumption we introduce throughout this section.
Assumption 10.
Suppose max( ∥ A ∥ , ∥ B ∥ , ∥ O ∥ ,
∥ Q ∥ , ∥ R ∥ ) ≤ C .
Suppose ∥ Ξ 1 h ( π ) ∥ ≤ 1 for any π .
O is full-column rank.
We present the form of linear mean embedding operators in LQGs.
Lemma 26 (Linear mean embedding operator) .
Let z ∈
Z h − 1 , o ∈ O , s ∈ S .
We have
E o
[Equation] [Equation]
0 O (cid:21) .


Proof.
Here, we have
E o ∼ O ( s )
[Equation]
oo
[Equation]
=
[Equation]
From the second line to the third line, we use formula vec[ A 1 A 2 A 3 ]
=
( A ⊤ 3 ⊗ A 1 )vec( A 2 ) .
This immediately concludes the result.
Thus, the matrix K h has the left inverse when O is full-column rank as follows:
[Equation] [Equation]
[Equation]
(cid:21)(cid:19)
[Equation]
I
O
[Equation]
(cid:21) ⊗ (cid:20) I
0 O
[Equation]
(cid:21) .
We use a block matrix inversion formula:
[Equation]
M.1 Existence of Link Functions
Lemma 27 (Value functions in LQGs) .
Let π h ( a | o, z )
=
δ ( a = U 1 h
o
+ U
2 h z ) for z ∈ Z h − 1 , o ∈
O .
Then, a value function has a bilinear form:
[Equation]
For any h ∈ [ H ] , these parameters Λ h , Γ h are recursively defined inductively by Λ H =
[Equation]
Γ
[Equation] [Equation]
Proof.
The proof is completed by backward induction regarding h , starting from level H .
First, we have
[Equation]
Here, we use induction.
Thus, supposing the statement is true at horizon h + 1 , we have
[Equation]
a )
[Equation]
] where z ′ is a vector that removes the last component ( o, a ) from z and s ′ is a state at h + 1 .
Here, recall we have [( z ′ )
⊤ , o ⊤ , a ⊤ , s ′⊤ ] ⊤ =
Ξ 1 h ( π )[ z ⊤ , s ⊤ ] ⊤ + Ξ 2 h
( π ) .
Then, the statement is concluded some algebra.
Lemma 28 (Norm constraints on value functions) .
We can set ∥ Λ h ∥ ≤ C Λ ,h , ∥ Γ h ∥ ≤ C Γ ,h such that
[Equation]
Proof.
We have
[Equation]
Then,
[Equation]
Since we assume ∥ Ξ 1 h ( π ) ∥ ≤ 1 , this immediately leads to
[Equation]
Besides, ∥
[Equation]
Thus, ∥
[Equation]
Next, we set the norm on the function class G h .
Lemma 29 (Realizability on LQGs) .
We set
[Equation]
A function class G h includes at least one value future-dependent value function for any linear policy π = δ ( a = U 1 h o + U 2 h z ) for ∥
U 1 h ∥ ≤ C , ∥ U 2 h ∥ ≤ C .
Proof.
Here, we have
[Equation]
The norm constraint on ¯Λ h is decided by the following calculation:
[Equation]
Then, the norm on ¯Γ h is decided by the following calculation:
[Equation]
From the first line to the second line, we use Lemma 43.
M.2 PO-bilinear Rank Decomposition
Lemma 30 (Bilinear rank decomposition for LQG) .
For any g h +1 ∈ G h
+1 , g h ∈
G h , we have the following bilinear rank decomposition:
[Equation]
where
[Equation]

[Equation]
Here
[Equation]
Ξ 1 h ( π ) and Σ Λ h 2 ( π ) depend on a policy π .
The following norm constraints hold:
[Equation]
Proof.
We have
[Equation]
Since we have
E [ r h ( z h − 1 , s h ); a 1: h − 1 ∼
[Equation]
′ , a h ∼
[Equation] [Equation]
we focus on the term (22).
Hereafter, we suppose the expectation is always taken under a 1: h − 1 ∼ π ′ , a ∼ π .
We also denote z =
z
h − 1 , o h
= o,
o h
+
= o ′ , s h
= s, s h
+1 =
s ′ to simplify the presentation.
Using this simplified notation, we get
[Equation]
Besides,
[Equation]
Then
[Equation]
the bilinear decomposition is clear by using
[Equation]
where A 2 is any vector and A 1 is any matrix.
First, we calculate the upper bounds of the norms.
[Equation]
F
[Equation] [Equation]
[Equation]
(cid:21)(cid:21)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
F
[Equation]
From the third line to the fourth line, we use ∥ Ξ 1 h ( π ) ∥ 2 ≤ 1 .
Thus, ∥ X h ( π ′ )
∥ 2 ≤ poly( H, d o , d s , d a , ∥
O † ∥ , C ) .
Next, we consider W ( π ) .
By some algebra, we can see
[Equation]
Lemma 31 (Variance of marginal distribution) .
Recall d πh ( z h − 1 , s h ) is a marginal distribution over Z h − 1 × S at h when we execute a 1: h − 1 ∼ π .
The distribution d πh ( z h − 1 , s h ) is a Gaussian distribution with mean 0 .
The operator norm on the variance of d πh ( z h − 1 , s h ) is upper-bounded by poly( H, d o , d a , d s , C ) .
Proof.
We first calculate the operator norm of the variance of d πh ( z h − 1 , o h ) .
The variance is h
[Equation]
i
[Equation]
The statement is immediately concluded.
Let u h (¯ z h , r h , a h , o h +1 ; θ )
=
θ ⊤ h ψ h (¯ z h ) − r h − θ ⊤
h +
ψ h +
(¯ z h +1 ) .
Recall ψ h (¯ z h ) =
[1 , ¯ z ⊤ h ⊗ ¯ z ⊤ h ] ⊤ .
We define
[Equation]
Then, the final estimator is constructed by
[Equation]
This is equal to
E
[Equation] [Equation]
π )] where l
[Equation]
I
[Equation]
∥
[Equation]
Z
[Equation]
[Equation] [Equation]
We set
[Equation]
for any i ∈ [3] .
M.3 Uniform Convergence
Recall that
[Equation]
C } .
Besides, G h is included in {⟨ θ, ψ h ( · ) ⟩ | ∥ θ ∥ ≤ poly( H, d o , d s , d a , C , ∥ O † ∥ ) } .
Lemma 32 (Concentration of loss functions) .
With probability 1 − δ , sup
[Equation]
∈ Π ,
[Equation] [Equation]
is upper-bounded by
[Equation]
Proof.
Due to indicator functions, l h (¯ z h , a h , o h , o h +1 ; θ, π ) is bounded for any π, θ by
[Equation]
Thus, for fixed π and θ , we can say that with high probability 1 − δ
[Equation]
Besides, we can consider a covering number with respect to l ∞ -norm for the space of K and θ since both are bounded.
The radius of each space is upper-bounded by
[Equation]
Thus, by taking uniform bound and considering the bias term due to the discretization as in the proof of Lemma 24, the statement is concluded.
Lemma 33 (Bias terms 1) .
Expectation of ˆ y h ( a [ i ] ) and ˆ y h ( a [0] ) are equal to y h ( a [ i ] )
+ Error 1 ,
y h
( a [0] )
+ Error 2 .
where
[Equation]
i α i ( π (¯ z h ))
} I ( ∥ ¯ z h ∥ ≤ Z 1 )
u h (¯ z h , a h , r h , o h +1 ; θ ); a 1: h − 1 ∼ π ′ , a h ∼ do (0) (cid:35) ,
Error 1 = m − 1 poly(ln( m ) , d s , d o , d a , C , H, ∥ O † ∥ ) ,
Error 2 = m − 1 poly(ln( m ) , d s , d o , d a , C , H, ∥ O † ∥ ) .
Proof.
We want to upper bound the difference of
[Equation]
and
[Equation]
By CS inequality
[Equation]
we have
[Equation]
[Table]
We analyze the term (a) and the term (b).
Before starting analysis, note (¯ z ⊤ h , a ⊤ h , r ⊤ h , o ⊤ h +1 ) follows
Gaussian distribution with mean 0 and variance upper-bounded by poly( C , d s , d o , d a , H ) using Lemma 31.
Besides, α 2 i ( π h (¯ z h )) ≤ poly( d s , d o , d a , H ) from Lemma 7.
Note we can use a
G-optimal design since we have a norm constraint on ¯
z 1 .
Regarding the term (a), by setting Z 2 = poly( C , d s , d o , d a , ln( m ) , H, ∥ O † ∥ ) and Z 3 = poly( C , d s , d o , d a , ln( m ) , H, ∥ O † ∥ ) properly, we can ensure it is upper-bounded by poly( C , d s , d o , d a , H, ∥
O † ∥ , ln( m )) m .
Regarding the term (b), noting high order moments of Gaussian distributions can be always upperbounded, the term (b) is upper-bounded by poly( C , d s , d o , d a , H, ∥
O † ∥ , ln( m )) .
This concludes the statement.
Lemma 34 (Bias terms 2) .
Recall we define y h ( a [ i ] ) and y h ( a [0] ) in Lemma 33.
Then, we have
[Equation]
i
[Equation]
Thus,
[Equation]
Proof.
First Statement We have
[Equation]
Here, by some algebra, there exists a vector c
[Equation]
h ] ⊗ [¯ z ⊤ h , s ⊤ h , a ⊤ h ]] ⊤ ⟩ .
Thus
[Equation]
there exists c 0 and a vector c 1 such that
[Equation]
Recall we can write
[Equation]
Using the above,
[Equation]
c 0 (¯ z h , s h ) (cid:35)(cid:35) + (cid:88) i
y h ( a [ i ] ) .
Besides, c 0 (¯ z h , s h ) =
E [ u h (¯ z h , a h , r h , o h +1 ; θ ) | ¯ z h , s h , a h
= do (0)] .
Thus,
E (cid:34)
I ( ∥ ¯ z h ∥ ≤ Z 1 ) (cid:34) c 0 (¯ z h , s h ) − (cid:88) i α i ( π h (¯ z h ))
c 0 (¯ z h , s h ) (cid:35)(cid:35) =
E [ I ( ∥ ¯ z h ∥ ≤ Z 1 ) { 1 − (cid:88) i
α i ( π h (¯ z h ))
} E [ u h (¯ z h , a h , r h , o h
+1 ; θ )
| ¯ z h , s h , a h = 0]; a 1: h − 1 ∼ π ′ ]
=
E [ I ( ∥ ¯ z h ∥ ≤ Z 1 ) { 1 − (cid:88) i
α i ( π h (¯ z h )) }
u h (¯ z h , a h , r h , o h +1 ; θ ); a 1: h − 1 ∼ π ′ , a h = do (0)]
= y h ( a [0] ) .
In conclusion,
E
[ I ( ∥ ¯ z h ∥ ≤ Z 1 )
u h (¯ z h , a h , r h , o h +1 ; θ ); a 1: h − 1 ∼ π ′ , a h ∼ π (¯ z j )]
= y h ( a [0] )
+ (cid:88)
i y h
( a [ i ] ) .
Second Statement As we see in the proof of Lemma 33, the following term
E [ { I ( ∥
[Equation]
Z
[Equation]
u h (¯ z
[Equation]
, a h , r h , o h +1 ; θ ); a 1: h − 1 ∼ π ′ , a h ∼ π (¯ z j )] is upper-bounded by poly( C , d s , d o , d a , H, ln( m )) /m.
Hence,
E [ l h (¯ z h , a h , r h , o h +1 ; θ, π )] =
[Equation]
a [0] )]
[Equation]
(Definition)
[Equation]
(Statement of Lemma 33) =
[Equation]
(First statement) =
[Equation] [Equation]
[Equation]
m
[Equation]
Br
[Equation] [Equation]
π
[Equation] [Equation]
d
[Equation]
d a
[Equation]
( m ))
[Equation]
m.
M.4 Sample Complexity
Summarizing results so far, we have
[Equation]
d s , d o , d a , C , H, ∥ O † ∥ ) ×
(cid:112) ln(1 /δ )
/m.
This is enough to invoke Theorem 1.
Here, recall we have ∥ X h ( π ) ∥
≤ poly( H, d o , d a , d s , C , Θ , ∥ O † ∥ ) , ∥
[Equation]
H
[Equation]
d
[Equation]
d a
[Equation]
d s , C , Θ , ∥ O † ∥ ) .
for any π ∈ Π using Lemma 30.
In addition, we showed the PO-bilinear rank is poly( H, d o , d a , d s ) .
Then, using Theorem 1, the sample complexity is
[Equation]
N Sample Complexity in PSRs
To focus on the main point, we just use a one-step future.
We first show the form of future-dependent value functions to set a proper class for G h .
Next, we show the PO-bilinear decomposition.
We assume the following assumptions.
Assumption 11.
(a) T ⊂ O is a core test and Q is a minimum core rest, (b) ∥ vec( J πh )
∥ ≤ Θ for any π ∈ Π where J πh is in V πh ( τ h ) =
1 ( z h − 1 )
⊤ J πh q τ h .
N.1 Existence of Link Functions
Recall V πh ( τ h ) =
1 ( z h − 1 )
⊤ J πh q τ h , where we use
( z ) ∈ R |O| M |A| M to denote the one-hot encoding vector over Z h − 1 , and J πh is a matrix in R |O| M |A| M ×|T | .
Then, g πh ( z h − 1 , o ) :
= 1 ( z h − 1 )
⊤ J πh [ 1 ( t = o )]
t ∈T is a value future-dependent value function.
This is because
[Equation]
Hence, we set G h to be
[Equation]
so that the realizability holds.
N.2 PO-bilinear Rank Decomposition
We show that PSR admits PO-bilinear rank decomposition (Definition 6).
Here is the Bellman loss:
[Equation]
To analyze the above, we decompose the above into three terms:
[Equation]
Let Q be a minimum core test.
Here, for any future t , there exists ˜ m t such that P ( t | τ h ) = ⟨ ˜ m t , ˜ q τ h ⟩ where [ P ( · | τ h )] |Q| is a |Q| -dimensional predictive state ˜ q τ h .
This satisfies
[Equation]
(23) where ˜ M o h ,a h is a matrix whose i -th row is
˜ m
⊤ o h ,a h as we see in Section E.
Term (c).
We have
[Equation]
where J 1 ∈ R |T |×|Q| is a matrix whose i -th row is ˜ m ⊤ t .
The existence of J 1 is ensured since Q is a core test.
Term (b).
We have
[Equation]
for some matrix J π 2 .
In the first inequality, we use the reward is a function of o h , a h conditioning on the whole history.
From the first line to the second line, we use a property of core tests.
Term (a).
We have
[Equation]
for some matrix J 3 .
Then, the above is further equal to (cid:88)
[Equation]
for some matrix J π 4 .
From the first line to the second line, we use P ( o h | τ h ; a h )˜ q τ h ,a h ,o h = ˜
M o h ,a h ˜ q τ h in (23).
Summary.
Combining all terms, there exists a matrix J π 5 such that
[Equation]
Here
[Equation]
we suppose ∥ Vec( J π 5 )
∥ ≤ Θ W for any π .
Besides,
[Equation]
Thus, we can set B X =
[Equation]
N.3 Sample Complexity
Suppose Π , G are finite and rewards at h lie in [0 , 1] .
Assume the realizability holds.
Then,
[Equation]
Following the calculation in Section I, the sample complexity is
[Equation]
Here, there is no explicit dependence on |T | .
Note the worst-case sample complexity of ln |G max | is
O ( |Z h − 1 ||T | ) and the worse-case sample complexity of ln | Π max | is O ( |Z h − 1 ||O||A| ) .
N.4 Most General Case
We consider the general case in Section E. Let G h be a function class consisting of 1 ( z h − 1 )
⊤ J h 1 ( t ) where J h satisfies
J h ∈ R Z
h − 1 ×|T | and ∥ vec( J h ) ∥ ≤ Θ .
When the realizability holds, we would get ˜
[Equation]
Here, there is no explicit sample complexity of |T O | .
Note the worse-case sample complexity of ln |G max | is O ( |Z h − 1 ||T | ) and the worst-case sample complexity of ln | Π max | is O ( |Z h − 1 ||O||A| ) .
O Proof of Theorem
We fix the parameters as in Theorem 8.
Let l h ( τ h , a h , r h , o h +1 ; f, π, g ) =
|A|
[Equation]
From the assumption, Then, with probability 1 − δ , we have ∀ t ∈
[ T ] , ∀ h ∈
[ H ] sup
[Equation] [Equation]
We first show the following lemma.
Recall π ⋆ = argmax π ∈ Π J ( π ) .
Lemma 35 (Optimism) .
Set R := ϵ gen .
For all t ∈ [ T ] , ( π ⋆ , g π ⋆ ) is a feasible solution of the constrained program.
Furthermore, we have J ( π ⋆ ) ≤ E [ g t 1 ( o 1 )]
+ 2 ϵ ini for any t ∈
[ T ] , where g t is the value link function selected by the algorithm in iteration t .
Proof.
For any π , we have
[Equation]
since g π is a value link function in G noting the condition (c) in Definition 9.
Thus,
[Equation]
using (24) noting π ⋆ ∈ Π , g π ⋆ ∈ G .
Hence, ( π ⋆ , g π ⋆ ) is a feasible set for any t ∈ [ T ] and any h ∈ [ H ] .
Then, we have
[Equation] [Equation]
(Uniform convergence result)
[Equation]
(Using the construction of algorithm)
[Equation]
(Uniform convergence)
Next, we prove the following lemma to upper bound the per step regret.
Lemma 36.
For any t ∈ [ T ] , we have
[Equation]
Proof.
[Equation]
(From (a) in Definition 3)
From Lemma 22, we have
[Equation]
Lemma 37.
[Equation]
Proof.
We have
[Equation]
The first term is upper-bounded by λB 2 W .
The second term is upper-bounded by t
[Equation]
[Equation]
From the first line to the second line, we use (b) in Definition 9.
From the second line to the third line, we use ξ is a non-decreasing function.
In the last line, we use the constraint on ( π t , g t ) .
Combining lemmas so far, we have
[Equation]
We set λ such that B 2 X /λ =
B 2 W B 2 X /ζ 2 ( ϵ gen )+
and
T = (cid:6) 2 Hd ln(4 Hd ( B 2 X B 2 W /ζ 2 (˜ ϵ gen ) + 1))
(cid:7) .
Then,
Hd
[Equation] [Equation]
≤since a ln( bT ) /T ≤ 1 when T = 2 a ln(2 ab ) .
Finally, the following holds
[Equation]
≤ H 1 / 2 (cid:2) 2 λB 2 W
+
Tζ 2 (2 ϵ gen ) (cid:3) 1 / 2 + 2 ϵ ini ≤ H 1 / 2 (cid:2) 2 λB 2 W + 2 ζ 2 (2 ϵ gen
) Hd ln(4 Hd ( B 2 X B 2 W /ζ 2 (˜ ϵ gen ) + 1)) (cid:3) 1 / 2 + 2 ϵ ini (Plug in T ) ≤ H 1 / 2 (cid:2) 4 ζ 2 ( ϵ gen ) + 2 ζ 2 (2 ϵ gen ) Hd ln(4 Hd ( B 2 X B 2 W /ζ 2 (˜ ϵ gen ) + 1))
(cid:3) 1 / 2 + 2 ϵ ini .
(Plug in λ )
P Sample Complexity in M -step Decodable POMDPs
We first give a summary of our results.
Then, we show that an M -step decodable POMDP is a
PO-bilinear rank model.
After showing the uniform convergence of the loss function with fast rates, we calculate the sample complexity.
Since we use squared loss functions, we need to modify the proof of Theorem 1.
P.1 PO-bilinear Rank Decomposition (Proof of Lemma 17)
In this section, we derive the PO-bilinear decomposition of M -step decodable POMDPs (Lemma).
First, we define moment matching policies following [19].
We denote M ( h ) =
h − M .
Definition 10 (Moment Matching Policies) .
For h ′ ∈
[ M ( h ) , h ] , we define
[Equation]
) ∈
X l where X l = S l × O l × A l − 1 and l = h ′ − M ( h ) + 1 .
For an M -step policy π and h ∈
[ H ] , we define the moment matching policy µ π,h
= { µ π,hh ′ :
X h ′ − M ( h )+
→
∆( A ) } h − 1 h ′ =
M ( h ) :
[Equation]
Note the expectation in the right hand side is taken under a policy π .
Using [19, Lemma B.2], we have
[Equation]
where
[Equation]
Thus
[Equation]
the first condition in Definition 9 ((12)) is satisfied
Next
[Equation]
we show the second condition in Definition 9 ((13)).
This is proved as follows
[Equation]
From the first line to the second line, we use [19, Lemma B.2].
From the third to the fourth line, we use the Bellman completeness assumption: − ( B πh G )
+ G h ⊂ F h .
From the fourth line to the fifth line, we use importance sampling.
Finally
[Equation]
we show the third condition in Definition 9 (14):
[Equation]
This follows since
[Equation]
) (cid:0) g πh (¯ z h ) − r h − g πh +1 (¯ z h +1 ) (cid:1) f (¯ z h ) − 0 .
5 f
(¯ z h ) 2 ; a 1: M ( h ) − 1 ∼ π ′ , a M ( h ): h ∼ U ( A ) (cid:3) =
E (cid:2) E
[ |A| π h ( a h | ¯ z h ) (cid:0) g πh (¯ z h ) − r
h − g πh +1 (¯ z h +1 ) (cid:1) | ¯ z h ] f (¯ z h ) − 0 .
5 f
(¯ z h ) 2 ; a 1: M ( h ) − 1 ∼ π ′ , a M ( h ): h ∼ U ( A ) (cid:3) =
E (cid:2) − 0 .
5 f
(¯ z h ) 2 ; a 1: M ( h ) − 1 ∼ π ′ , a M ( h ): h ∼ U ( A ) (cid:3) .
P.2 Uniform Convergence
We define the operator ( B πh g
[Equation]
(¯ z h
[Equation]
:=
E [ r h + g h +
(¯ z h +
[Equation]
| ¯ z h
[Equation]
a h ∼ π ] .
and
[Equation]
Lemma 38 (Uniform Convergence) .
Let |D| = m .
Suppose ∥F h ∥ ∞ ≤ 3 H for h ∈ [ H ] .
Fix π ′ ∈ Π .
Take a true future-dependent value function g π ∈ G .
Then, it satisfies
[Equation]
m .
Suppose g ( π ) satisfies
[Equation]
[Table]
[Equation]
and the Bellman completeness ¯ B πh G ⊂ F h ( ∀ π ∈ Π) holds.
Then, with probability 1 − δ , we have
[Equation]
Proof.
To simplify the notation
[Equation]
we define
[Equation]
Given g ∈ G
[Equation]
we define ˆ f h ( · ; g ) as the maximizer:
[Equation]
In this proof
[Equation]
the expectation is always taken for the data generating process D .
We first observe
[Equation]
Then, we define
Er
[Equation]
As the first step
[Equation]
we prove with probability 1 − δ
[Equation]
We first fix g .
Then, from the definition of ˆ f h ( · ; g ) and the Bellman completeness ¯ B πh G ⊂ F h , we have
[Equation]
Here, we invoke Bernstein’s inequality:
[Equation]
Hereafter
[Equation]
we condition on the above event.
Then, combining (30) and (31)
[Equation]
we have
[Equation]
Here, we use
E
[Equation]
Thus, by some algebra,
[Equation]
Besides, | E
[Equation]
Lastly, by union bounds over G h , G h +1 , the statement (29) is proved.
Note ¯ B πh g π = 0 .
First Statement.
[Equation]
From the second line to the third line, we use (29).
Second Statement.
Now, we use the assumption on g ( π ) :
[Equation]
From what we showed in (29)
[Equation]
this implies
[Equation]
Recall we want to upper-bound the error for E [0 .
( ¯ B πh g ( π )) 2 (¯ z h )] .
Here, we use the following observation later:
[Equation]
We use Bernstein’s inequality: with probability 1 − δ , for any g ∈
[Equation]
Here, we use
[Equation]
Hereafter, we condition on the above event.
Finally
[Equation]
we have
[Equation]
g ( π ))
(¯ z h ) − 0 .
5( B πh g ( π )) 2 (¯ z h )] | ≤ Λ + 3(12 H |A| )
2 ln(4 |F h ||G h ||G
h +1 | /δ ) m
+ (cid:114) E
[0 . 5( B πh g ( π ))
(¯ z h )]
ln(4 |G
h ||G h +1 | /δ )
m
+ ln( |G h ||G h +
| /δ ) m .
Hence,
[Equation]
P.3 Proof of Main Statement
We define
[Equation]
Let
[Equation]
Then, from the first statement in Lemma 38, with probability 1 − δ , ∀ t ∈ [ T ] , ∀ h ∈ [ H ] , ∀ π ∈ Π
[Equation]
Besides, from the second statement in Lemma
[Equation]
for π ∈ Π
[Equation]
∀ t ∈ [ T ]
[Equation]
∀ h ∈
[ H ] , when g ( π ) satisfies
[Equation]
we have
[Equation] [Equation]
We first show the optimism.
Recall π ⋆ = argmax π ∈ Π J ( π ) .
Lemma 39 (Optimism) .
Set R =
ϵ 2 gen .
For all t ∈ [ T ] , ( π ⋆ , g π ⋆ ) is a feasible solution of the constrained program.
Furthermore, we have J ( π ⋆ ) ≤ E [ g t 1 ( o 1 )]
+ 2 ϵ ini for any t ∈ [ T ] .
Proof.
For any π ∈ Π , letting g π ∈ G be a corresponding value future-dependent value function
[Equation]
we have
[Equation]
using (32).
This implies
[Equation]
Hence, ( π ⋆ , g π ⋆ ) is a feasible set for any t ∈ [ T ] .
Then, we have
[Equation]
Next, recall the following two statements.
The following statements are proved as before in the proof of Theorem 1.
• For any t ∈ [ T ] ,
[Equation]
|⟨ W h
( π t , g t ) , X h ( π t ) ⟩| + 2 ϵ ini .
• Let Σ t,h = λI
+ (cid:80) t − 1 τ =
X h ( π τ ) X h ( π τ ) ⊤ .
We have
[Equation]
Lemma 40.
[Equation]
Proof.
We have
[Equation]
The first term is upper-bounded by λB 2 W .
The second term is upper-bounded by t − 1 (cid:88) τ =
[Equation]
From the first line to the second line, we use (26).
Here, from the third line to the fourth line, we use (33).
The rest of the argument is the same as the proof in Theorem 1.
Finally, the following holds
[Equation]
Sample Complexity Result.
We want to find m such that
[Equation]
where
[Equation]
By organizing terms
[Equation]
we have
[Equation]
Thus
[Equation]
setting the following m is enough:
[Equation]
The total sample we use mTH is
[Equation]
Q Sample Complexity in Observable POMDPs with Latent Low-rank Transition
This section largely follows the one in Section P.
Q.1 Existence of Value Future-Dependent Value Functions
Since we consider the discrete setting, we can set the value future-dependent value function class as
Section K. Hence, we set
[Equation]
Then, we can ensure ∥G h ∥ ≤ H/σ 1 .
Then, from the construction of F h , we can also ensure ∥F h ∥ ≤ 4 H/σ 1 .
Q.2 PO-bilinear Rank Decomposition (Proof of Lemma 18)
In this section
[Equation]
we derive the PO-bilinear decomposition of observable POMDPs with the latent low-rank transition.
We want to prove Lemma 18.
Recall M ( h ) = max( h − M, 1) .
Using [
[Equation]
Lemma B.2]
[Equation]
we have
[Equation]
where
[Equation]
( s M ( h ) − 1 , a M ( h ) − 1 ); a 1: M ( h ) − 1 ∼ π ′ ] .
Thus, the first condition in Definition 9 is satisfied
Next
[Equation]
we show the second condition in Definition 9.
This is proved as follows:
[Equation]
From the first line to the second line, we use [19, Lemma B.2].
From the third to the fourth line, we use the Bellman completeness assumption: − ( B πh G )
+ G h ⊂ F h .
From the fourth line to the fifth line, we use importance sampling.
The third condition
[Equation]
is easily proved.
Finally, the following norm constraints hold:
[Equation]
Sample Complexity Result.
Following the same procedure as Section P, here, we want to find m such that
[Equation]
where
[Equation]
By organizing terms, we have
[Equation]
Thus, setting the following m is enough
[Equation]
The total sample we use mTH is
[Equation]
Finally, we plug-in ln( | Π max ||F max ||G max | /δ )
=
ln( |M| ) .
R Exponential Stability for POMDPs with Low-rank Transition
In this section, we prove that the short memory policy is a globall near optimla policy in low-rank
MDPs.
We first introduce several notation.
Next, we prove the exponential stability of Bayesian fileters, which immediately leads to the main statement.
Notation.
Given a belief b ∈ ∆( S ) , an action and observation pair ( a, o ) , we define the Bayesian update as follows.
We define B ( b, o ) ∈ ∆( S ) as the operation that incorporates observation o , i.e., b ′ =
B ( b, o ) with b ′ ( s ) =
O ( o | s ) b ( s ) / ( (cid:80) ¯ s O ( o | ¯ s ) b (¯ s )) , and T a b as the operation that incorporates the transition, i.e., ( T a b )( s ′ )
= (cid:80) s b ( s ) T ( s ′ | s, a ) .
Finally, we denote U ( b, a, o ) as the full Bayesian filter, i.e.,
[Equation]
Let us denote b 0 ∈ ∆( S ) as the initial latent state distribution.
Given the first observation o 1 ∼ O ( ·| s ) , s ∼ b 0 , we denote b 1 = B ( b 0 , o 1 ) as the initial belief of the system conditioned on the first observation o 1 .
Given two beliefs b, b ′ , we define the distance D 2 ( b, b ′ ) :
=
log E s ∼ b [ b ( s ) /b
( s ′ )]
Consider a POMDP whose latent transition is low rank, i.e., T ( s ′ | s, a ) = µ ( s ′ )
⊤ ϕ
( s, a ) .
For notation simplicity, we still consider discrete state, action, and observation space to avoid using measure theory languages.
Design of initial distribution.
We want to design a good distribution for the initial distribution in an artificial Bayesian filter ignoring the history other than the short history.
The following lemma is from [23, Lemma 4.9] that quantifies the contraction of a Bayesian map.
Lemma 41 (Contraction propery of beliefs) .
Suppose b, b ′ ∈ ∆( S ) and ∥ b/b ′ ∥ ∞ < ∞ .
Then we have:
[Equation]
Next, we compute the G-optimal design using feature ϕ ( s, a ) : S ×
A → R .
Denote the G-optimal design as ρ ∈ ∆( S × A ) .
Here, we use assumption ∥
ϕ ( s, a ) ∥ ≤ 1 for any ( s, a ) in Assumption 6, which ensures that ϕ ( s, a ) lives in a compact space for any ( s, a ) .
The property is given as in
Theorem 7.
In summary, the support of ρ (denoted by S ρ ) is at most d ( d + 1) / 2 points and for any ϕ
( s, a ) , there exists α ( s, a ) such that
[Equation]
[Equation]
where we denote the points on the support S ρ as { s i , a i } | S ρ | i =1 .
We set our “empty" belief as follows:
[Equation]
Note that this belief ˜ b 0 does not depend on any history.
We aim to bound D 2 ( b, ˜ b 0 ) using the following lemma where b is some belief resulting from applying T a for any a to a belief ˜ b ∈ ∆( S ) .
This is a newly introduce lemma.
Lemma 42 (Distance between the actual belief and the designed initial distribution) .
For any distribution b ∈ ∆( S ) that results from a previous belief ˜ b and a one-step latent transition under action a , i.e., b ( s ) =
T a ˜ b
˜ s ) , we have:
[Equation]
Proof.
For any b ∈ ∆( S ) , using its definition, we have:
[Equation]
(Property of G-optimal design)
[Equation]
Similarly, the construction of ˜ b 0 implies that ˜ b 0 ( s ) =
(cid:80) S
ρ i =1 ρ ( s i , a i ) ϕ
( s i , a i ) ⊤ µ ( s ) , thus, we have:
[Equation]
(Use propety of G-optimal design(34))
Thus, D
[Equation]
Now we prove the exponential stability by leveraging Lemma 42 and Lemma 41.
Theorem 14 (Exponential stability for POMDPs with Low-rank Latent Transition) .
Consider a t ≥
Cγ − 4 ln( d/ϵ ) .
Consider any policy (full history dependent) π and a trajectory a 1: h + t − 1 , o 1: h + t ∼ π for h ≥ 1 .
Denote b h + t as the (true) belief conditioned on a 1: h + t − 1 , o 1: h + t .
For approximated belief, first for h = 1 , we define ¯ b h + t as:
[Equation]
for h ≥ 2 , we define ¯ b h + t as:
[Equation]
Then we have:
[Equation]
Proof.
We define
[Equation]
Hereafter
[Equation]
we omit ( o 1: h + n , a 1: h + n − 1 ) to simplify the notation.
We start from the base case Y h
(i
[Equation]
e., n = 0 ).
First case, consider h > 1 , b h
=
U ( b h − 1 , o h , a h − 1 ) .
Denote b ′ h
=
T
a h − 1 b h − 1 .
From Lemma 42, we know that:
[Equation]
Thus
[Equation]
noting b h
= B ( b ′ h , o h ) and ¯ b h = B (
˜ b h , o h )
[Equation]
we have:
[Equation]
which implies the base case:
[Equation]
Now for any n ≥
[Equation]
we have:
[Equation]
a 1: h − 1 ; a 1: h + n − 1 ∼ π ] = E (cid:20)(cid:113)
[Equation]
o
[Equation]
h
[Equation] [Equation]
h
[Equation]
a
[Equation]
h
[Equation]
n
[Equation]
(cid:21) ≤ (1 − σ 41 / 2 40 ) E (cid:20)(cid:113) exp (cid:0) D
[Equation]
(cid:0) ( T a h
[Equation]
n
[Equation]
1 b h + n − 1 ) , ( T a h + n − 1 ¯ b h + n − 1 ) (cid:1) / 4 (cid:1) − 1 | o 1: h − 1 , a 1: h − 1 ; a 1: h + n − 1 ∼ π (cid:21) ≤ (1 − σ 41 / 2 40 ) E (cid:20)(cid:113) exp (cid:0)
[Equation]
(cid:0)
[Equation]
h
[Equation]
n
[Equation]
1 , ¯
[Equation]
h
[Equation]
n
[Equation]
1 ) (cid:1)
[Equation]
(cid:1) −
[Equation]
o 1: h
[Equation] [Equation]
a
[Equation]
h
[Equation] [Equation]
a 1: h + n − 1 ∼ π (cid:21) (Data processing inequality from [23, Lemma 2.7])
[Equation]
This completes the induction step.
Adding expectation with respect to the history a 1: h − 1 , o 1: h −back, we conclude the proof.
When h = 1 , we simply start with the original belief b 1 .
For any 0 ≤ n ≤ t , we simply set ¯ b 1+ n = b 1+ n , thus the conclusion still holds.
The above Theorem 14 indicates that in order to approximate the ground truth belief b h + t that is conditioned on the entire history, we only need to apply the Bayesian filter on the M memory ¯ z h + t starting from a fixed distribution ˜ b 0 .
The existence of such ˜ b 0 is proven by construction where we rely on the low-rankness of the latent transition and a D-optimal design over S ×
A using the feature ϕ .
The above Theorem 14 together with the proof of Theorem 1.2 in [23] immediately implies for
M = Θ( C ( σ 1 ) − 4 ln
( dH/ϵ ))
(with C being some absolute constant), there must exists an M-memory policy π ⋆ , such that J ( π ⋆gl ) − J ( π ⋆ ) ≤ ϵ – thus a globally optimal policy can be approximated by a policy that only relies on short memories.
S Auxiliary Lemmas
We use the following in Section 4.2.
Lemma 43 (Useful inequalities) .
• ∥ AB ∥ ≤ ∥
A ∥∥ B ∥ , ∥ AB ∥ F ≤ ∥
A
[Equation]
B ∥ F vec( aa ⊤ ) = a ⊗ a, ∥ vec( A ) ∥ 2 = ∥
A ∥ F , Tr( AB ) = vec( A ⊤ ) ⊤ vec( B ) .
• When A and B are semi positive definite matrices, we have
[Equation]
The following lemma is useful when we calculate the sample complexity.
Lemma 44.
The following is satisfied
[Equation]
when
[Equation]
for some constant c .