S AMPLE C OMPLEXITY OF N ONPARAMETRIC O FF -P OLICY E VALUATION ON L OW -D IMENSIONAL M ANIFOLDS USING D EEP N ETWORKS
A BSTRACT
We consider the off-policy evaluation problem of reinforcement learning using deep convolutional neural networks.
We analyze the deep fitted Q-evaluation method for estimating the expected cumulative reward of a target policy, when the data are generated from an unknown behavior policy.
We show that, by choosing network size appropriately, one can leverage any low-dimensional manifold structure in the Markov decision process and obtain a sample-efficient estimator without suffering from the curse of high data ambient dimensionality.
Specifically, we establish a sharp error bound for fitted Q-evaluation, which depends on the intrinsic dimension of the state-action space, the smoothness of Bellman operator, and a function class-restricted χ 2 -divergence.
It is noteworthy that the restricted χ 2 -divergence measures the behavior and target policies’ mismatch in the function space , which can be small even if the two policies are not close to each other in their tabular forms.
We also develop a novel approximation result for convolutional neural networks in Q-function estimation.
Numerical experiments are provided to support our theoretical analysis.
I NTRODUCTION
Off-policy Reinforcement Learning (RL) [38, 40] is an important area in decision-making applications, when the data cannot be acquired with arbitrary policies.
For example, in clinical decisionmaking problems, experimenting new treatment policies on patients is risky and may raise ethical concerns.
Therefore, we are only allowed to generate data using certain policies (or sampling distributions), which have been approved by medical professionals.
These so-called “behavior policies” are unknown but could impact our problem of interest resulting in distribution shift and insufficient data coverage of the problem space.
In general, the goal is to design algorithms that need as little data as possible to attain desired accuracy.
A crucial problem in off-policy RL is policy evaluation.
The goal of Off-Policy Evaluation (OPE) is to estimate the value of a new target policy based on experience data generated by existing behavior policies.
Due to the mismatch between behavior and target policies, the off-policy setting is entirely different from the on-policy one, in which policy value can be easily estimated via Monte Carlo.
A popular algorithm to solve OPE is the fitted Q-evaluation method (FQE), as an off-policy variant of the fitted Q-iteration [28, 15, 75].
FQE iteratively estimates Q-functions by supervised regression using various function approximation methods, e.g., linear function approximation, and has achieved great empirical success [65, 20, 21], especially in large-scale Markov decision problems.
Complementary to the empirical studies, several works theoretically justify the success of FQE.
Under linear function approximation, [31] show that FQE is asymptotically efficient, and [15] further provide a minimax optimal non-asymptotic bound, and [47] provide a variance-aware characterization of the distribution shift via a weighted variant of FQE.
[75] analyze FQE with realizable, general differentiable function approximation.
[37, 64] tackle OPE for even more general function approximation, but they require stronger assumptions such as full data coverage.
[16] focus on on-policy estimation and study a kernel least square temporal difference estimator.
Recently, deploying neural networks in FQE has achieved great empirical success, which is largely due to networks’ superior flexibility of modeling in high-dimensional complex environments
Nonetheless, the theory of FQE using deep neural networks has not been fully understood.
While there are existing results on FQE with various function approximators [28, 15, 75], many of them are not immediately applicable to neural network approximation.
[18] focus on the online policy learning problem and studies DQN with feed-forward ReLU network; a concurrent work [73] studies offline policy learning with realizable, general differentiable function approximation.
Notably, a recent study [51] provide an analysis of the estimation error of nonparametric FQE using feed-forward ReLU network, yet this error bound grows quickly when data dimension is high.
Moreover, their result requires full data coverage, i.e., every state-action pair has to eventually be visited in the experience data.
Precisely, besides universal function approximation, there are other properties that contribute to the success of neural networks in supervised learning, for example, its ability to adapt to the intrinsic low-dimensional structure of data .
While these properties are actively studied in the deep supervised learning literature, they have not been reflected in RL theory.
Hence, it is of interest to examine whether these properties still hold in a problem with sequential nature under standard assumptions and how neural networks can take advantage of such low-dimensional structures in OPE.
Main results.
This paper establishes sample complexity bounds of deep FQE using convolutional neural networks (CNNs).
Different from existing results, our theory exploits the intrinsic geometric structures in the state-action space.
This is motivated by the fact that in many practical highdimensional applications, especially image-based ones [59, 11, 76], the data are actually governed by a much smaller number of intrinsic free parameters [2, 55, 32].
See an example in Figure 1.
Consequently, we model the state-action space as a d -dimensional Riemannian manifold embedded in R D with d ≪ D .
Under some standard regularity conditions, we show CNNs can efficiently approximate Q-functions and allow for fast-rate policy value estimation—free of the curse of ambient dimensionality D .
Moreover, our results do not need strong data coverage assumptions.
In particular, we develop a function class-restricted χ 2 -divergence to quantify the mismatch between the visitation distributions induced by behavior and target policies.
The function class can be viewed as a smoothing factor of the distribution mismatch, since the function class may be insensitive to certain differences in the two distributions.
Our approximation theory and mismatch characterization significantly sharpen the dimension dependence of deep FQE.
In detail, our theoretical results are summarized as follows:
(I)
Given a target policy π , we measure the distribution shift between the experience data distribution { q data h } Hh =1 and the visitation distribution of target policy { q πh } Hh
[Equation]
by
[Equation]
where χ 2 Q ( q πh , q data h ) is the restricted χ 2 -divergence between q πh and q data h defined as
[Equation]
with Q being a function space relevant to our algorithm.
(II)
We prove that the value estimation error of a target policy
π
is
[Equation]
where K is the effective sample size of experience data sampled by the behavior policy (more details in Section 3), H is the length of the horizon, α is the smoothness parameter of the Bellman operator, κ is defined in (1), and (cid:101) O ( · ) hides some constant depending on state-action space and a polynomial factor in D .
We compare our results with several related works in Table 1.
Both [15] and [75] consider parametric function approximation to the Q-function: [15] study linear function approximation, and [75] assume third-order differentibility, which is not applicable to neural networks with non-smooth activation.
On the other hand, [51] use feed-forward neural networks with ReLU activation to parametrize nonparametric Q-functions, but they do not take any low-dimensional structures of the state-action space into consideration.
Therefore, their result suffer from the curse of dimension D .
Moreover, they characterize the distribution shift with the absolute density ratio between the experience data distribution and the visitation distribution of target policy, which is strictly larger than our characterization in restricted χ 2 -divergences.
As can be seen from our comparison, our result improves over existing results.
Moreover, since CNNs have been widely used in deep RL applications and also retain state-of-the-art performance [48, 22], our consideration of CNNs is a further step of bridging practice and theory.
[Table]
O (cid:0) H
[Equation]
α/ (
[Equation]
+2 D
[Equation]
K
[Equation] [Equation]
D
[Equation]
This work
Besov
CNN
[Equation]
O
[Equation]
Additional Related Work Besides FQE, there are other types of methods in the OPE literature.
One popular type is using importance sampling to reweigh samples by the distribution shift ratio [56], but importance sampling suffers from large variance, which is exponential in the length of the horizon in the worst case.
To address this issue, some variants with reduced variance such as marginal importance sampling (MIS) [68] and doubly robust estimation [35, 61] have been developed.
For the tabular setting with complete data coverage, [72] show that MIS is an asymptotically efficient OPE estimator, which matches the Cramer-Rao lower bound in [35].
Moreover, a line of work [49, 50, 39, 74] focuses on policy evaluation without function approximation using MIS and linear programming.
Notation For a scalar a > 0 , ⌈ a ⌉ denotes the ceiling function, which gives the smallest integer which is no less than a ; ⌊ a ⌋ denotes the floor function, which gives the largest integer which is no larger than a .
For any scalars a and b , a ∨ b denotes max( a, b ) and a ∧ b denotes min( a, b ) .
For a vector or a matrix, ∥·∥ 0 denotes the number of nonzero entries and ∥·∥ ∞ denotes the maximum magnitude of entries.
Given a function f : R D
→ R and a multi-index s =
[ s 1 , · · · , s D ] ⊤ , ∂ s f denotes ∂ | s | f ∂x s 11 ··· ∂x sDD .
∥
f ∥ L p denote the L p norm of function f .
We adopt the convention 0 /
= 0 .
Given distributions p and q , if p is absolutely continuous with respect to q , the Pearson χ 2 -divergence is defined as χ 2 ( p, q ) :
=
E q [( d p d q − 1) 2 ] .
P RELIMINARIES 2. T IME - INHOMOGENEOUS M ARKOV D ECISION P ROCESS
We consider a finite-horizon time-inhomogeneous
Markov
Decision
Process (MDP) ( S , A , { P h } Hh =1 , { R h } Hh =1 , H, ξ ) , where ξ is the initial state distribution.
At time step h = 1 , · · · , H , from a state s in the state space S , we may choose action a from the action space A and transition to a random next state s ′ ∈ S according to the transition probability distribution s ′ ∼ P h ( · | s, a ) .
Then, the system generates a random scalar reward r ∼ R h ( s, a ) with r ∈ [0 , 1] .
We denote the mean of R h ( s, a ) by r h ( s, a ) .
A policy π = { π h } Hh =1 specifies a set of H distributions π h ( · | s ) for choosing actions at every state s ∈ S and time step h .
Given a policy π , the state-action value function, also known as the
Q-function , for h = 1 , 2 , · · · , H , is defined as
[Equation]
where a h ′ ∼ π h ′ ( · | s h ′ ) and s h
′ +1 ∼ P h ′ ( · | s h ′ , a h ′ ) .
Moreover, let q πh denote the state-action visitation distribution of π at step h , i.e., q πh ( s, a ) :
= P π [ s h
= s, a h = a | s 1 ∼ ξ ] .
For notational ease, we denote X := S × A .
Let P πh : R X → R X denote the conditional transition operator at step h : P
[Equation]
where a ′ ∼ π h ( · | s ′ ) and s ′ ∼ P h ( · | s, a ) .
Denote the Bellman operator at time h under policy π as T πh :
T πh f ( s, a
[Equation]
:= r h
[Equation]
+ P [Equation] , ∀ f : X → R .
The Bellman equation may be written as Q πh = T πh Q πh +1 .
2. R IEMANNIAN M ANIFOLD
Let M be a d -dimensional Riemannian manifold isometrically embedded in R D .
A chart for M is a pair ( U, ϕ ) such that U ⊂ M is open and ϕ : U → R d is a homeomorphism, i.e., ϕ is a bijection, its inverse and itself are continuous.
Two charts ( U, ϕ ) and ( V, ψ ) are called C k compatible if and only if
[Equation]
and ψ ◦ ϕ
[Equation]
1 : ϕ
[Equation]
U ∩ V ) → ψ ( U ∩ V ) are both C k functions ( k -th order continuously differentiable).
A C k atlas of M is a collection of C k compatible charts { ( U i , ϕ i ) } such that (cid:83)
i U i =
M .
An atlas of M contains an open cover of M and mappings from each open cover to R d .
Definition 1 (Smooth manifold) .
A manifold M is smooth if it has a C ∞ atlas.
We introduce the reach [19, 53] of a manifold to characterize the curvature of M .
Definition 2 (Reach, Definition 2.1 in [1]) .
The medial axis of M is defined as T ( M ) , which is the closure of
T ( M ) = { x ∈ R D | ∃ x 1 ̸ = x 2 ∈ M , such that ∥ x − x 1 ∥ 2 = ∥ x − x 2 ∥ 2 = inf y ∈M ∥ x − y ∥ 2 } .
The reach ω of M is the minimum distance between M and T ( M ) , i.e.
[Equation]
Roughly speaking, reach measures how fast a manifold “bends”.
A manifold with a large reach “bends” relatively slowly.
On the contrary, a small ω signifies more complicated local geometric structures, which are possibly hard to fully capture.
2. B ESOV F UNCTIONS ON S MOOTH M ANIFOLD
Through the concept of atlas, we are able to define Besov space on a smooth manifold.
Definition 3 (Modulus of Smoothness [12]) .
Let Ω ⊂ R D .
For a function f : R D → R be in L p (Ω) for p > 0 , the r -th modulus of smoothness of f is defined by
[Equation]
p [Equation] f, t ) = sup ∥ h ∥ 2 ≤ t ∥
∆ rh ( f ) ∥
L p , where
[Equation]
rj =0 (cid:0) rj (cid:1) ( − 1) r − j f ( x + jh ) if x ∈ Ω , x + rh ∈ Ω , otherwise .
Definition 4 (Besov functions on M ) .
Let M be a compact manifold of dimension d with a finite atlas { ( U i , ϕ i ) } .
A function f : M (cid:55)→ R belongs to the Besov space B αp,q ( M ) , if on any chart U i , it holds
[Equation]
where
[Equation]
Further, the Besov norm of f is defined as ∥ f ∥ B αp,q ( M )
=
(cid:80) i ∥ f
∥ B αp,q ( U i ) .
We occasionally omit M in the Besov norm when it is clear from the context.
2. C ONVOLUTIONAL N EURAL N ETWORK
We consider one-sided stride-one convolutional neural networks (CNNs) with rectified linear unit (ReLU) activation function ( ReLU( z ) = max( z, 0) ).
Specifically, a CNN we consider consists of a padding layer, several convolutional blocks, and finally a fully connected output layer.
Given an input vector x ∈ R D , the network first applies a padding operator P : R D
→ R D × C for some integer C ≥ 1 such that
[Equation]
Then the matrix Z is passed through M convolutional blocks.
We will denote the input matrix to the m -th block as Z m and its output as Z m +
(i.e., Z 1 = Z ).
I
I × C matrix for the j -th output channel.
Let us define the convolution operation in
Equation (2).
Let W =
W j,i,l } ∈ R C ′
×
I × C be a filter where C ′ is the output channel size, I is the filter size and C is the input channel size.
For Z ∈ R D × C , the convolution of W with Z , denoted with W ∗ Z , results in Y ∈ R D × C ′ with
Y k,j =
I (cid:88) i =
C (cid:88) l =
[Equation]
where we set Z k + i − 1 ,l = 0 for k
+ i − 1 > D .
In the m -th convolutional block, let W m =
{W (1) m , ..., W ( L m ) m } and B m =
{ B (1) m , ..., B ( L m ) m } be a collection of filters and biases of proper sizes.
The m -th block maps its input matrix Z m ∈ R D × C to Z m
+1 ∈ R D × C by
[Equation]
with ReLU applied entrywise.
For notational simplicity, we denote this series of operations in the m -th block with a single operator from R D × C to R D × C with Conv W m , B m , so (2) can be abbreviated as
[Equation]
Overall, we denote the mapping from input x to the output of the M -th convolutional block as
[Equation]
Given (3), a CNN applies an additional fully connected layer to G and outputs
[Equation]
where W ∈ R D × C and b ∈ R are a weight matrix and a bias, respectively, and ⊗ denotes sum of entrywise product, i.e. W ⊗ G ( x ) =
(cid:80) i,j W i,j [ G ( x )] i,j .
Thus, we define a class of CNNs of the same architecture as
[Equation]
[Equation]
b with ∥ W ∥
[Equation]
∨ | b | ≤ τ 2 , G ( x ) in the form of (3) with M blocks.
The number of filters per block is bounded by L ; filter size is bounded by I ; the number of channels is bounded by J ; max m,l
[Equation]
Furthermore, F ( M, L, J, I, τ 1 , τ 2 , V ) is defined as
[Equation]
Universal Approximation of Neural Networks There exists rich literature about using neural networks to approximate functions supported on compact domain in Euclidean space, from early asymptotic results [34, 23, 10, 33] to later quantitative results [3, 46, 44, 30, 71].
These results suffer from the curse of dimensionality, in that to approximate a function up to certain error, the network size grows exponentially in the data dimension.
Recently, a line of work advances the approximation theory of neural network to functions supported on domains with intrinsic geometric structures [8, 57, 60, 58, 7, 42].
They show that neural networks are adaptive to the intrinsic structures in data, suggesting that to approximate a function up to certain error, it suffices to choose the network size only depending on the intrinsic dimension, which is often much smaller than the representation dimension of the data.
In addition to existing results, we prove a novel universal approximation theory of CNN en route to our RL result.
N EURAL F ITTED Q-E VALUATION
We consider the off-policy evaluation (OPE) problem of a finite-horizon time-inhomogeneous MDP.
The transition model { P h } H h =1 and reward function { R h } H h =1 are unknown, and we are only given the access to an unknown behavior policy π 0 to generate experience data from the MDP.
Our objective is to evaluate the value of π from a fixed initial distribution ξ over horizon H , given by
[Equation]
where a h ∼ π ( · | s h ) and s h +1 ∼ P h ( · | s h , a h ) .
At time-step h , we generate data D h :
= { ( s h,k , a h,k , s ′ h,k , r h,k ) }
Kk =1 .
Specifically, { s h,k } Kk =1 are i.i.d.
samples from some state distribution.
For each s h,k , we use the unknown behavior policy π 0 to generate a h,k ∼ π 0 ( · | s h,k ) .
More generally, we may view { ( s h,k , a h,k ) } Kk =1 as i.i.d.
samples from a sampling distribution q π 0 h .
For each ( s h,k , a h,k ) , we can further generate s ′ h,k ∼ P h ( · | s h,k , a h,k ) and r h,k ∼ R h ( s h,k , a h,k ) independently for each k .
This assumption on data generation is the time-inhomogeneous analog of a standard data assumption in time-homogeneous OPE, which assumes all data are i.i.d.
samples from the same sampling distribution [51, 74, 50, 69].
Moreover, our dataset is similar to one that comprises K independent episodes generated by the behavior policy, as [41] introduce a subroutine whereby one can process an episodic dataset and treat it as an i.i.d.-sampled dataset in any downstream algorithm.
To estimate v π , neural FQE estimates Q πh in a backward, recursive fashion.
(cid:98) Q πh , our estimate at step h , is taken as (cid:98) T πh (cid:16) (cid:98) Q πh [Equation] 1 (cid:17) , whose update rule is based on the Bellman equation: [Equation]
[Equation]
where (cid:98) T πh is an intermediate estimate of the Bellman operator T πh , (cid:98) Q πh +1 is an intermediate estimate of Q πh +1 , and F denotes a class of convolutional neural networks as specified in (5) with proper hyperparameters.
The pseudocode for our algorithm is presented in Algorithm 1.
M AIN R ESULTS
In this section, we prove an upper bound on the estimation error of v π by Algorithm 1.
First, let us state two assumptions on our MDP of interest.
Assumption 1 (Low-dimensional state-action space) .
The state-action space X is a d -dimensional compact Riemannian manifold isometrically embedded in R D , i.e., X ⊆ M .
There exists B >such that ∥ x ∥ ∞ ≤ B for any x ∈ X .
The reach of X is ω > 0 .
Assumption 1 characterizes the low-dimensional structures of the MDP represented in high dimensions.
We say that the “ intrinsic dimension ” of X is d ≪ D .
Such a setting, as mentioned in Section 1, is common in practice, because the representation or feature people have access to are often excessive compared to the latent structures of the problem.
For instance, images of a dynamical system are widely believed to admit such low-dimensional latent structures [26, 32, 55].
People often take the visual display of a computer game as its state representations, which are in pixels, but computer only keeps a small number of parameters internally to represent the state of the game.
Assumption 2 (Bellman completeness) .
Under target policy π , for any time step h and any f ∈ F , we have T πh f ∈ B αp,q ( X ) , where 0 < p, q ≤ ∞ and d/p + 1 ≤ α < ∞ .
Moreover, there exists a constant c 0 > 0 that satisfies ∥T πh f ∥ B αp,q ( X ) ≤ c 0 for any time step h .
Bellman completeness assumption is about the closure of a function class under the Bellman operator.
It has been widely adopted in RL literature [70, 14, 6, 69].
Some classic MDP settings implicitly possess this property, e.g., linear MDP [36].
Note that [66] show the necessity of such an assumption on the Bellman operator to regulate the Bellman residual: without such assumption, even in the simple setting of linear function approximation with realizability, to solve OPE up to a constant error, the lower bound on sample complexity is exponential in horizon.
The Besov family contains a large class of smooth functions, and has been widely adopted in existing nonparametric statistics literature for various problems [24, 62, 63].
For MDP, Assumption 2 holds for most common smooth dynamics, as long as certain regularity conditions on smoothness are satisfied.
For instance, [51] show a simple yet sufficient condition, under which for any time step h and s ′ ∈ S , the reward function r h ( s, a ) and the transition kernel P h ( s ′ | s, a ) are functions in
B αp,q .
This further implies Q π 0 , · · · , Q πH ∈ B αp,q ( X ) .
In addition, Assumption 2 may be satisfied even when the transition kernel is not smooth, examples of which are provided in [18].
Note that while most existing work on function approximation assumes Bellman completeness with respect to the function approximator, which in our work is deep convolutional neural networks with
ReLU activation, we are only concerned with the closure of the Besov class B αp,q ( X ) under the
Bellman operator.
This assumption is weaker than the previous work [75], which considers smooth function approximation (excluding ReLU networks).
Our main result is summarized in Theorem 2, which relies on using CNNs to accurately represent
T π h f for any f ∈ F .
The following theorem provides a novel quantatitive analysis on how to properly choose CNN classes for approximating T πh f depending on the regularity of T πh .
Theorem 1.
Suppose Assumption 1 and 2 hold.
For any positive integers I ∈ [2 , D ] and (cid:102)
M, (cid:101) J > 0 , we let
[Equation] [Equation]
Then CNN class F ( M, L, J, I, τ 1 , τ 2 ) in (4) can approximate T πh f for any f ∈ F ( M, L, J, I, τ 1 , τ 2 ) and h = 1 , · · · , H , i.e., there exists (cid:98) f ∈ F ( M, L, J, I, τ 1 , τ 2 ) with
[Equation]
O ( · ) hides a constant depending on d , α , 2 d αp − d , p , q , ∥T πh f ∥ B αp,q ( X ) , B , ω and surface area of X .
The proof is provided in Appendix B. As can be seen, the rate of approximation is free of the curse of ambient dimension D .
We remark that Theorem 1 allows an arbitrary rescaling of (cid:102) M and (cid:101) J , as only their product is relevant to the approximation error.
This is more flexible than conventional approximation theories [7, 54, 51], where the network width and depth have to maintain a fixed ratio in terms of the desired approximation error.
In Theorem 2, we choose a configuration of (cid:102) M and (cid:101) J that leads to the optimal statistical rate via a bias-variance tradeoff argument.
Theorem 2.
Suppose Assumption 1 and 2 hold.
By choosing
[Equation]
with any integer I ∈ [2 , D ] in Algorithm 1, in which O ( · ) hides factors depending on d , α , 2 d αp − d , p , q , c 0 , B , ω , and the surface area of X , we have
[Equation]
in which the expectation is taken over the data, and C is a constant depending on D
[Equation] [Equation]
d , d
[Equation]
α
[Equation]
d αp − d , p , q , c 0 , B , ω and the surface area of X .
The distributional mismatch is captured by
[Equation]
in which q πh and q π 0 h are the visitation distributions of π and π 0 at step h respectively and Q is the Minkowski sum between the CNN function class in (5) and the Besov function class, i.e., Q = {
f + g | f ∈ B αp,q ( X ) , g ∈ F ( M, L, J, I, τ 1 , τ 2 , V ) } .
We next compare our Theorem 2 with existing work:
(I) Tight characterization of distributional mismatch .
The term κ depicts the distributional mismatch between the target policy’s visitation distribution and data coverage via restricted χ 2divergence.
Note that the restricted χ 2 -divergence is always no
larger than the commonly-used absolute density ratio [51, 6, 67] and can often be substantially smaller.
This is because probability measures q πh and q π 0 h might differ a lot over some small regions in the sample space, while their integrations of a smooth function in Q over the entire sample space could be close to each other.
The absolute density ratio measures the former and restricted χ 2 -divergence measures the latter.
More strikingly, when considering function approximation (e.g. state-action space is not countably finite), the restricted χ 2 -divergence can still remain small even when absolute density ratio becomes unbounded.
For example, we consider two isotropic multivariate Gaussian distributions with different means.
[52] has shown that Pearson χ 2 -divergence, which is always larger than or equal to restricted χ 2 -divergence, has a finite expression:
[Equation]
whereas one may find the absolute density ratio unbounded: for any µ 1 ̸ = µ 2 ,
[Equation]
Such a stark comparison can also be observed in other common distributions that have support with infinite cardinality, e.g. Poisson distribution.
Furthermore, when the state-action space exhibits small intrinsic dimensions, i.e., d ≪ D , the restricted χ 2 -divergence adapts to such low-dimensional structure and characterizes the distributional mismatch with respect to Q , which is a small function class depending on the intrinsic dimension.
In contrast, the absolute density ratio in [51] does not take advantage of the low-dimensional structure.
In summary, though the absolute density ratio is a tight in the tabular setting and some other special classes of MDPs, in
the general function approximation setting, it could easily become intractably vacuous, and restricted χ 2 -divergence is tighter characterization of distributional mismatch.
(II) Adaptation to intrinsic dimension .
Note that our estimation error is dominated by the intrinsic dimension d , rather than the representation dimension D .
Therefore, it is significantly smaller than the error of methods oblivious to the problem’s intrinsic dimension such as [51].
Such a fast convergence owes to the adaptability of neural networks to the manifold structure in the state-action space.
With properly chosen width and depth, the neural network automatically captures local geometries on the manifold through the empirical risk minimization in Algorithmfor approximating Q-functions.
Sample Complexity Comparison .
Given a pre-specified estimation error of policy value ϵ , our algorithm requires a sample complexity of
[Equation]
We next compare our result with [51], which among existing work is the most similar to ours.
Specifically, we reprove our result with feed-forward ReLU network so as to be in the same setting as [51] (details in Theorem 3 of Appendix E).
When the experience data are allowed to be reused, they show a sample complexity of
[Equation]
As can be seen, our result is more efficient than theirs as long as H 1 − D − dα ≤ ϵ − 1 − D − d/ 2 α .
Such a requirement of the horizon can be satisfied in real applications, as d ≪ D and α is moderate.
Note that even with no consideration for low-dimensional structures, i.e., d = D , our result is still more efficient, as κ is often substantially smaller than κ 3 .
Moreover, when the experience data are used just for one pass, our method is instantly more efficient, as their sample complexity becomes
[Equation]
E XPERIMENTS
We present numerical experiments for evaluating FQE with CNN function approximation on the classic CartPole environment [4].
The CartPole problem has a 4 -dimensional continuous intrinsic state space.
We consider a finite-horizon MDP with horizon H = 100 in this environment.
In our experiments, we solve the OPE problem with FQE (Algorithm 1).
We take the visual display of the environment as states.
These images serve as a high-dimensional representation of CartPole’s original 4 -dimensional continuous state space.
In our algorithm, we use a deep CNN to approximate the Q-functions and solve the regression with SGD (see Appendix F.1 for details).
65 . 2 is computed via Monte Carlo rollout.
[Table]
We consider two settings with different visual resolutions (see Appendix F.1 for details): one in high resolution (dimension 3 × 40 × 150 ) and the other in low resolution (dimension 3 × 20 × 75 ).
We use a policy trained for 200 iterations with REINFORCE as the target policy.
We conduct this experiment in two cases: (A) data are generated from the target policy itself; (B) data are generated from a mixture policy of 0 .
8 target policy and 0 . 2 uniform distribution.
(A) aims to verify the performance’s dependence on data intrinsic dimension without the influence from distribution shift.
We observe that the performance of FQE on high-resolution and low-resolution data is similar, in both the off-policy case and the easier case with no distribution shift.
It shows that the estimation error of FQE takes little influence from the representation dimension of the data but rather from the intrinsic structure of the environment, which is the same regardless of resolution.
We also observe that the estimation becomes increasingly accurate as sample size K increases.
These empirical results confirm our upper bound in Theorem 2, which is only dominated by data intrinsic dimension.
C ONCLUSION
This paper studies nonparametric off-policy evaluation in MDPs.
We use CNNs to approximate Qfunctions.
Our theory proves that when state-action space exhibits low-dimensional structures, the finite-sample estimation error of FQE converges depending on the intrinsic dimension.
In the estimation error, the distribution mismatch between the data distribution and target policy’s visitation distribution is quantified by a restricted χ 2 -divergence term, which is oftentimes much smaller than the absolute density ratio.
Our theory also reassures practitioners of the benignity of overrepresentation in deep RL and provides insights into how to choose network hyperparameters properly in presence of low intrinsic dimension.
We support our theory with experiments.
For future directions, it would be of interest to adapt this low-dimensional analysis to time-homogeneous MDPs.
It is nontrivial to preserve the error rate with sample reuse in the presence of temporal dependency.
A CKNOWLEDGEMENT
Mengdi Wang acknowledges the support by NSF grants DMS-1953686, IIS-2107304, CMMI1653435, and ONR grant 1006977.
Tuo Zhao acknowledges the support by DMS-2012652.
A P ROOF OF T HEOREM
In this section
[Equation]
we provide a proof for the upper bound on the estimation error in Theorem 2.
We can tackle the sequential dependencies by recursively conditioning on the previous-step estimation and the fact that the error in the previous steps accumulates linearly.
The estimation error can be decomposed into a sum of statistical error and approximation error.
A tradeoff exists about the network size: while a larger network reduces the approximation error, it leads to higher variance in the statistical error.
Consequently, we choose the network size and architecture appropriately to balance the two types of error, which in turn minimizes the final estimation error.
Proof of Theorem 2.
The goal is to bound
[Equation]
To get an expression for that
[Equation]
we first expand it recursively.
To illustrate the recursive relation, we examine the quantity at step h :
[Equation]
[Table]
[Equation]
where C denotes a (varying) constant depending on D 3 α 2 α + d , d , α , d αp − d , p , q , c 0 , B , ω and the surface area of X .
In (a), note T πh (cid:98) Q πh +1 ∈ B αp,q ( X ) by Assumption 2 and − (cid:98) T πh (cid:16) (cid:98) Q πh +1 (cid:17) ∈ F by our algorithm, so
T π h (cid:98) Q π h +1 − (cid:98) T
π h (cid:16) (cid:98) Q
π h +1 (cid:17) ∈ Q .
Then we employ a change-of-measure argument and obtain this inequality by invoking the following lemma.
Lemma 1.
Given a function class Q that contains functions mapping from X to R and two probability distributions q 1 and q 2 supported on X , for any g ∈ Q ,
[Equation]
Proof of Lemma 1.
[Equation]
where the last step is by the definition of χ
[Equation]
q
[Equation] [Equation]
=
[Equation]
In (b), we use Jensen’s inequality and the fact that square root is concave.
To obtain (c), we invoke Lemma 10, which provides an upper bound on the error of nonparametric regression at each step of the FQE algorithm.
Specifically, we will invoke Lemma 10 when conditioning on D h +1 , · · · , D H , i.e. the data from time step h + 1 to time step H .
Note that after conditioning, T πh (cid:98) Q πh +1 becomes measurable and deterministic with respect to D h +1 , · · · , D H .
Also, D h +1 , · · · , D H are independent from D h , which we use in the regression at step h .
To justify our use of this theorem, we need to cast our problem into a regression problem described in the theorem.
Since { ( s h,k , a h,k ) } Kk =1 are i.i.d.
from q π 0 h , we can view them as the samples x i ’s in the lemma.
We can view T πh (cid:98) Q πh +1 , which is measurable under our conditioning, as f 0 in the lemma.
Furthermore, we let
[Equation]
( s
[Equation] [Equation]
[Equation] [Equation]
s
[Equation] [Equation]
[Equation]
In order to invoke Lemma 10 under the conditioning on D h +1 , · · · , D H , we need to verify whether three conditions are satisfied (conditioning on D h +1 , · · · , D H ):
Sample { ( s h,k , a h,k ) }
Kk =1 are i.i.d; 2.
Sample { ( s h,k , a h,k ) }
Kk =1 and noise { ζ h,k } Kk =1 are uncorrelated; 3.
Noise { ζ h,k } Kk =1 are independent, zero-mean, subgaussian random variables.
In our setting, { ( s h,k , a h,k ) } Kk =1 are i.i.d.
from q π 0 h .
Due to the time-inhomogeneous setting, they are independent from D
h
+1 , · · · , D H , so { ( s h,k , a h,k ) }
Kk =1 are still i.i.d.
under our conditioning.
Thus, Condition 1 is clearly satisfied.
We may observe that under our conditioning, the transition from ( s h,k , a h,k ) to s ′ h,k is the only source of randomness in ζ h,k , besides ( s h,k , a h,k ) itself.
The distribution of ( s h,k , a h,k , s ′ h,k ) is actually the product distribution between P h ( ·| s h,k , a h,k ) and q π 0 h , so a function of s
′ h,k , generated from the transition distribution P h ( ·| s h,k , a h,k ) , is uncorrelated with ( s h,k , a h,k ) .
Thus, ( s h,k , a h,k ) ’s are uncorrelated with ζ h,k ’s under our conditioning, and Condition 2 is satisfied.
Condition 3 can also be easily verified.
Under our conditioning, the randomness in ζ h,k only comes from ( s h,k , a h,k , s ′ h,k , r h,k ) , which are independent from ( s h,k ′ , a h,k ′ , s ′ h,k ′ , r h,k ′ ) for any k ′ ̸
=
k , so ζ h,k ’s are independent from each other.
As for the mean of ζ h,k ,
E [ ζ h,k | D h +1 , · · · , D H ]
[Equation]
=
+ 0 = 0 .
On the other hand, (cid:13)(cid:13)(cid:13) (cid:98) Q πh +1 (cid:13)(cid:13)(cid:13) ∞ ≤ H almost surely, because it is a function in our CNN class F .
Thus, ζ h,k is a bounded random variable with ζ h,k ∈ [ − 2 H, 2 H ] almost surely, so its variance is bounded by 4 H 2 .
Its boundedness also implies it is a subgaussian random variable.
Thus, Condition 3 is also satisfied.
Hence, Lemma 10 proves, for step h in our algorithm,
[Equation]
where C ′ depends on D
α
α + d , d , α , 2 d αp − d , p , q , c 0 , B , ω and the surface area of X .
Note that this upper bound holds for any (cid:98) Q πh +1 or D h
+1 , · · · , D H .
The sole purpose of our conditioning is that we could view (cid:98) Q πh +1 as a measurable or deterministic function under the conditioning and then apply Lemma 10.
Therefore,
[Equation]
Finally, we carry out the recursion from time step 1 to time step H , and the final result is
[Equation]
B P ROOF OF T HEOREM
For simplicity, let us denote f 0 :
=
T πh f in the theorem statement.
Note that f 0 ∈ B αp,q ( X ) .
Moreover, let us define a class of single-block CNNs in the form of
[Equation]
as
F
[Equation] [Equation]
f | f
[Equation]
x ) in the form of (3) with L layers.
The number of filters per block is bounded by L ; filter size is bounded by I ; the number of channels is bounded by J ; max m
[Equation]
l ∥W
[Equation]
We will refer to CNNs in this form as “single-block CNNs” and use them as building blocks of our final CNN approximation for the ground truth Besov function.
B. P ROOF O VERVIEW OF T HEOREM
Theorem 1 serves as a building block for Theorem 2, which establishes the relation between network architecture and approximation error.
For simplicity, denote c 0 :=
∥ f 0 ∥ B αp,q ( X ) .
Theorem 1 is proven in the following steps:
[Equation]
AS
SUM OF LOCALLY SUPPORTED FUNCTIONS OVER MANIFOLD
Since manifold X is assumed compact (Assumption 1), we can cover it with a finite set of Ddimensional open Euclidean balls { B β ( c i ) } C X i =1 , where c i denotes the center of the i -th ball and β is its radius.
We choose β < ω/ 2 , and define U i =
B β ( c i )
∩ X .
Note that each U i is diffeomorphic to an open subset of R d (Lemma 5.4 in Niyogi et al.
[53]); moreover, { U i } C X i =1 forms an open cover for X .
There exists a carefully designed open cover with cardinality C X ≤ ⌈ A ( X )
β d T d ⌉ , where A ( X ) denotes the surface area of X and T d denotes the thickness of U
i ’s, i.e., the average number of U
i ’s that contain a given point on X .
T d is O
( d log d ) (Conway et al.
Moreover, for each U i , we can define a linear transformation
[Equation]
where a i ∈ R is the scaling factor and b i ∈ R
d is the translation vector, both of which are chosen to ensure ϕ
( U i ) ⊂ [0 , 1] d , and the columns of V i ∈ R D ×
d form an orthonormal basis for the tangent space T c i ( X ) .
Overall, the atlas { ( ϕ i , U i ) } C X i =1 transforms each local neighborhood on the manifold to a d -dimensional cube.
Thus, we can decompose f 0 using this atlas as
[Equation]
12) because there exists such a C ∞ partition of unity { ρ i } C X i =1 with supp( ϕ i ) ⊂
U i (Proposition 1 in Liu et al.
Since each f i is only supported on U i , we can further write
[Equation] [Equation]
where 1 U i is the indicator for membership in U i .
Lastly, we extend f i ◦ ϕ − 1 i to entire [0 , 1] d with 0 , which is a function in B αp,q ([0 , 1] d ) with
B αp,q ([0 , 1] d ) Besov norm at most Cc 0 (Lemma 4 in Liu et al.
[42]), where C is a constant depending on α , p , q and d .
This extended function is to be approximated with cardinal B-splines in the next step.
S TEP 2: A PPROXIMATE EACH LOCAL FUNCTION WITH CARDINAL B- SPLINES
With most things connected with the intrinsic dimension d in the last step, we proceed an approximation of f 0 on the low-dimensional manifold.
With α ≥ d/p + 1 assumed in Assumption 2, we can invoke a classic result of using cardinal B-splines to approximate Besov functions (Lemma 5), by setting r = + ∞ and m =
⌈ α ⌉ + 1 in the lemma.
It states that there exists a weighted sum of cardinal B-splines (cid:101)
f i in the form (cid:101) f
i ≡
N (cid:88)
j = (cid:101)
f i,j ≈ f i ◦ ϕ − 1 i with (cid:101) f
i,j = c ( i ) k, j (cid:101) g
d k, j ,m (14) such that
[Equation]
In (14), c ( i ) k, j ∈ R is coefficient and (cid:101) g dk, j ,m : [0 , 1] d → R denotes a cardinal B-spline with index k, m ∈ N + , j ∈ R d .
k is a scaling factor, j is a shifting vector, m is the degree of the B-spline.
By (13) and (14), we now have a sum of cardinal B-splines
[Equation]
which can approximate our target Besov function f 0 with error
[Equation]
S TEP 3: A PPROXIMATE EACH CARDINAL B- SPLINE WITH A COMPOSITION OF CNN S
Each summand in (16) is a composition of functions, each of which we can implement with a CNN.
Specifically, we do so with a special class of CNNs defined in (11), which we refer to as ”singleblock CNNs”.
The multiplication operation × can be approximated by a single-block CNN (cid:98) × with at most η error in the L ∞ sense (Proposition 1).
(cid:98) × needs O (log 1 η ) layers and 6 channels.
All weight parameters are bounded by ( c 20 ∨ 1) .
We consider each (cid:101)
f i ◦ ϕ
i together, which we can approximate with a sum of N CNNs (cid:98) f SCNN i,j ◦ (cid:98) ϕ
i up to δ error, namely,
[Equation]
≤ δ
[Equation]
In particular, we can use a single-block CNN (cid:98) f SCNN i,j to approximate the B-spline (cid:101)
f i,j up to δ/N error.
Moreover, since ϕ i is linear, it can be expressed with a single-layer perceptron (cid:98) ϕ i .
The architecture and size of (cid:98) f SCNN i,j and (cid:98) ϕ
i are characterized in Proposition 2 as functions of δ .
1 U i is an indicator for membership in U i , so we need 1 U
i ( x ) = 1 if d 2 i ( x ) =
∥ x − c i ∥ 22 ≤ βand 1 U i ( x ) = 0 otherwise.
By this definition, we can write 1 U i as a composition of a univariate indicator 1 [0 ,β 2 ] and the distance function d 2 i :
[Equation]
Given θ ∈ (0 , 1) and ∆ ≥ 8 DB 2 θ , it turns out that 1 [0 ,β 2 ] and d 2 i can be approximated with two single-block CNNs (cid:98) 1 ∆ and (cid:98) d 2 i respectively (Proposition 3) such that
[Equation]
and
[Equation]
some value between 0 and 1 , otherwise .
[Equation]
The architecture and size of (cid:98) 1 ∆ and (cid:98) d 2 i are characterized in Proposition 3 as functions of θ and ∆ .
The above three approximations rely on the classic result of using CNN to approximate cardinal
B-splines (Lemma 10 in Liu et al.
[42]; Lemma 1 in Suzuki [60]).
Putting the above together, we can develop a composition of single-block CNNs
[Equation] [Equation]
as an approximation for (cid:101) f i,j ◦ ϕ i × 1 U i .
The overall approximation error of ¯ f i,j can be written as a sum of the three types of approximation error above.
Details are provided in Appendix B.2.
Moreover, by Lemma 6, there exists a single-block CNN (cid:98) f i,j that can express ¯ f i,j .
S TEP 4: E XPRESS THE SUM OF CNN COMPOSITIONS WITH A CNN
Finally, we can assemble everything into (cid:98) f
[Equation]
which serves as an approximation for f 0 .
By choosing the appropriate network size in Lemma 2, which the tradeoff between the approximation error of (cid:98) f i,j and its size, we can ensure that
[Equation]
By Lemma 7, for (cid:102) M, (cid:101) J > 0 , we can write this sum of N · C X single-block CNNs as a sum of (cid:102) M single-block CNNs with the same architecture, whose channel number upper bound J depends on (cid:101) J .
This allows Theorem 1 to be more flexible with network architecture.
By Lemma 4, this sum of (cid:102)
M CNNs can be further expressed as one CNN in the CNN class (5).
Finally, N will be chosen appropriately as a function of network architecture parameters, and the approximation theory of
CNN is proven.
When Theorem 1 is applied in our problem setting, we will take the target function f 0 above to be
T πh (cid:98) Q πh +1 at each time step h , which is the ground truth of the regression at each step of Algorithm 1.
For more details about this, please refer to the proof of Theorem 2 in Appendix A.
B. P ROOF OF T HEOREM
In the following, we provide the proof details for Theorem 1, which quantifies the tradeoff between a CNN in the class of 11 and its approximation error for Besov functions on a low-dimensional manifold.
We start from the decomposition of the approximation error of (cid:98) f , which is based on the decomposition of the approximation error of ¯ f i,j in (21), and will proceed to the end of this proof.
Lemma 2.
Let η be the approximation error of the multiplication operator (cid:98) × ( · , · ) as defined in Step 3 of Appendix B.1 and Proposition 1, δ be defined as in Step 3 of Appendix B.1 and Proposition 2, ∆ and θ be defined as in Step 3 of Appendix B.1 and Proposition 3.
Assume N is chosen according to Proposition 2.
For any i = 1 , ..., C X , we have (cid:13)(cid:13)(cid:13) (cid:98) f − f 0 (cid:13)(cid:13)(cid:13) L ∞ ≤ (cid:80) C X
i
=
( A i, 1 + A i, 2 + A i, 3 ) with
[Equation]
for some constant C ′′ depending on d, α, p, q and some constant c .
Furthermore, for any ε ∈ (0 , 1) , setting
[Equation]
gives rise to
[Equation]
The choice in (24) satisfies the condition ∆ > 8 B 2 Dθ in Proposition 3.
Proof of Lemma 2.
As in Proposition 1, A i, 1 measures the error from (cid:98) × :
[Equation]
i, 1 =
[Equation]
for some constant C ′′ depending on d, α, p, q .
The last inequality is due to the choice of N in
Proposition 2.
A i, 2 measures the error from CNN approximation of Besov functions.
As in Proposition 2, A i, 2 ≤ δ .
A i, 3 measures the error from CNN approximation of the chart determination function.
The bound of A i, 3 can be derived using the proof of Lemma 4.5 in Chen et al.
[7], since f i ◦ ϕ − 1 i is a Lipschitz function and its domain is in [0 , 1] d .
In order to attain the error desired in Lemma 2, we need each network in ¯ f i,j with appropriate size.
The network size of the components in ¯ f i,j can be analyzed as follows:
• (cid:98)
i : The chart determination network (cid:98) 1 i = (cid:98) d 2 i ◦ (cid:98) 1 ∆ is the composition of (cid:98) d 2 i and (cid:98) 1 ∆ .
By Proposition 3, (cid:98) d 2 i is a single-block CNN with O (log 1 θ ) =
O ( αd log N + D + log D ) layers and width 6 D ; (cid:98) 1 ∆ is a single-block CNN with O (log( β 2 / ∆))
=
O ( αd log N ) layers and width 2 .
In both subnetworks, all parameters are of O (1) .
By Lemma 6, the chart determination network (cid:98) 1 i is a single-block CNN with O ( αd log N + D + log D ) layers, width 6 D + 2 and all weight parameters are of O (1) .
• (cid:98) ×
By Proposition 1, the multiplication network is a single-block CNN with O (log 1 η ) =
O ((1 + αd ) log N ) layers and O (1) width.
All weight parameters are bounded by ( c 20 ∨ 1) .
• (cid:98) ϕ
i : The projection ϕ
i is a linear one, so it can be expressed with a single-layer perceptron.
By Lemma 8 in Liu et al.
[42], this single-layer perceptron can be expressed with a singleblock CNN with 2 + D layers and width d .
All parameters are of O (1) .
• (cid:98) f
SCNN i,j : by Proposition 2, each (cid:98) f
SCNN i,j is a single-block CNN with O (log 1 δ )
=
O ( αd log N ) layers and ⌈ 24 d ( α + 1)( α + 3) + 8 d ⌉ channels.
All weight parameters are in the order of O (cid:16)
[Table]
[Equation]
Next, we want to show ¯ f i,j , a composition of the aforementioned single-block CNNs, can be simply expressed as a single-block CNN.
By Lemma 6, there exists a single-block CNN g i,j with O (log N + D ) layers and ⌈ 24 d ( α + 1)( α + 3) + 9 d ⌉ width realizing (cid:98)
f SCNN i,j ◦ (cid:98) ϕ
i .
All weight parameters in g i,j are in the order of
O (cid:16) N (log2) αd ( 2 d αp − d + c 1 d − 1 ) (cid:17) .
Moreover, recall that the chart determination network (cid:98) 1 i is a singleblock CNN with O (log N + D + log D ) layers and width 6 D + 2 , whose weight parameters are of O (1) .
By Lemma 14 in Liu et al.
[42], one can construct a convolutional block, denoted by ¯ g i,j , such that
[Equation]
Here ¯ g i,j has ⌈ 24 d ( α + 1)( α + 3) + 9 d ⌉ + 6 D + 2 channels.
Since the input of (cid:98) × is (cid:20) g
i,j (cid:98)
i (cid:21) , by Lemma 15 in Liu et al.
[42], there exists a CNN ˚
g
i,j which takes (25) as the input and outputs (cid:98) × ( g i,j , (cid:98) 1 i ) .
Note that ¯ g i,j only contains convolutional layers.
The composition ˚ g
i,j ◦ ¯ g i,j , denoted by (cid:98) g SCNN i,j , is a CNN and for any x ∈ X ,
(cid:98) g SCNN i,j ( x ) =
¯ f i,j ( x ) .
We have (cid:98) g SCNN i,j ∈ F SCNN ( L, J, I, τ, τ ) with
L =
O (log N + D + log D ) , J = ⌈ 48 d ( α + 1)( α + 3) + 18 d ⌉ + 12 D
+ O
[Equation] [Equation]
O
[Equation]
N
[Equation]
, (26) and I can be any integer in [2 , D ] .
Therefore, we have shown that (cid:98) g SCNN i,j is a single-block CNN that expresses ¯ f i,j , as we desired.
Furthermore, recall that (cid:98) f can be written as a sum of C X N such SCNNs.
By Lemma 7, for any (cid:102) M, (cid:101) J satisfying (cid:102) M (cid:101) J
= O ( N ) , there exists a CNN architecture F SCNN ( L, J, I, τ, τ ) that gives rise to a set of single-block CNNs { (cid:98) g i } (cid:102)
Mi =1 ∈ F SCNN ( L, J, I, τ, τ ) with (cid:98) f
= (cid:102) M (cid:88)
i =1 (cid:98) g i (27) and
L =
O (log N + D + log D ) , J =
O ( D (cid:101) J ) , τ
= O (cid:16) N (log2) dα ( 2 d
αp − d + c 1 d − 1 ) (cid:17) .
By Lemma 3 below, we slightly adjust the CNN architecture by re-balancing the weight parameter boundary of the convolutional blocks and that of the final fully connected layer.
In particular, we rescale all parameters in convolutional layers of (cid:98) g i to be no larger than 1.
While this procedure does not change the approximation power of the CNN, it can make the CNN have a smaller covering number, which is conducive to a smaller variance.
Lemma 3 (Lemma 16 in Liu et al.
Let γ ≥ 1 .
For any g ∈ F SCNN ( L, J, I, τ 1 , τ 2 ) , there exists f ∈ F SCNN ( L, J, I, γ − 1 τ, γ L τ ) such that g ( x ) =
f
( x ) .
In this case, we set γ = c ′ N (log2) dα ( 2 d αp − d + c 1 d − 1 ) (8 ID ) (cid:102) M 1 L , where c ′ is a constant such that τ ≤ c ′ N (log2) dα (
[Equation]
d
[Equation]
d + c 1 d − 1 ) .
With this γ , we have (cid:98) f i ∈ F SCNN ( L, J, I, τ 1 , τ 2 ) with
[Equation] [Equation]
Finally, we prove that it suffices to use one CNN to realize the sum of single-block CNNs in (27).
Lemma 4.
Let F SCNN ( L, J, I, τ 1 , τ 2 ) be any CNN architecture from R D to R .
Assume the weight matrix in the fully connected layer of F SCNN ( L, J, I, τ 1 , τ 2 ) has nonzero entries only in the first row.
For any positive integer M , there exists a CNN architecture F ( M, L, J, I, τ 1 , τ 2 (1 ∨ τ − 1 1 )) such that for any { (cid:98)
f i ( x ) } Mi =1 ⊂ F SCNN ( L, J, I, τ 1 , τ 2 ) , there exists (cid:98) f ∈ F ( M, L, 4 + J, I,
[Equation]
1 , τ
[Equation]
)) with
[Equation]
Consequently, by Lemma 4, there exists a CNN that can express our sum of (cid:102)
M single-block CNNs with architecture F ( M, L, J, I, τ 1 , τ 2 ) with
[Equation]
29) and (cid:101) J, (cid:102)
M satisfying
[Equation] [Equation]
which is a requirement inherited from Lemma 7.
This CNN is our final approximation for f 0 .
Applying this relation N =
O ( (cid:102) M (cid:101) J ) to (29) gives
[Equation]
and the network size
[Equation]
B. P ROOF OF L EMMA
Denote the architecture of (cid:98) f m with (cid:98)
[Equation]
, where W m = (cid:110) W ( l ) m (cid:111)
L l =1 , B m = (cid:110) B ( l )
m (cid:111)
L l =1 .
Furthermore, denote the weight matrix and bias in the fully connected layer of (cid:98) f with (cid:99) W, (cid:98) b and the set of filters and biases in the m -th block of (cid:98) f with (cid:99) W m and (cid:98) B m , respectively.
The padding layer (cid:98) P in (cid:98) f pads the input x from R D to R D × 4 with zeros.
Each column denotes a channel.
Let us first show that for each m , there exists some Conv (cid:99) W m , (cid:98) B m : R D × 4 → R D × 4 such that for any
Z ∈ R D × 4 with the form
Z =
[( x ) + ( x ) − ⋆ ⋆ ] , (32) where ( x ) + means applying ( · ∨ 0) to every entry of x and ( x ) − means applying − ( · ∧ 0) to every entry of x , so all entries in Z are non-negative.
We have
[Equation]
where ⋆ ’s denotes entries that do not affect this result and may take any different value.
For any m , the first layer of f m takes input in R D .
Thus, the filters in W (1) m are in R D .
Again, we pad these filters with zeros to get filters in R D × 4 and construct (cid:99) W (1) m such that
[Equation]
where we use the fact that W (1) m ∗ ( x ) + − W (1) m ∗ ( x ) − = W (1)
m ∗ x .
The first four output channels at the end of this first layer is a copy of Z .
For the filters in later layers of (cid:98) f m and all biases, we simply set
[Equation]
for l = 2 , . . .
, L, ( (cid:99) W ( l ) m ) 2 , : , : = [ 0 e0 0 · · · 0 ] for l = 2 , . . .
, L, ( (cid:99) W ( l ) m ) 3 , : , : =
[ 0 0 e0 · · · 0 ] for l = 2 , . . .
, L − 1 , ( (cid:99) W ( l ) m ) 4 , : , : = [ 0 0 0 e· · · 0 ] for l = 2 , . . .
, L − 1 , ( (cid:99) W ( l ) m ) 4+ j, : , : = (cid:104) 0 0 0
( W ( l ) m ) j, : , : (cid:105) for l = 2 , . . .
, L − 1 , ( (cid:98) B ( l ) m ) j, : , : = (cid:104) 0 0 0
( B ( l ) m ) j, : , : (cid:105) for l = 1 , . . .
, L − 1 .
In Conv (cid:99) W m , (cid:98) B m , an additional convolutional layer is constructed to realize the fully connected layer in (cid:99) f m .
By our assumption, only the first row of W m is nonzero.
Furthermore, we set (cid:98) B ( L ) m = 0 and (cid:99) W Lm as size-one filters with three output channels in the form of ( (cid:99) W ( L ) m ) 3 , : , : = (cid:2) 0
e τ 1 τ 2 ( W m ) 1 , : (cid:3) , ( (cid:99) W ( L ) m ) 4 , : , : = (cid:2) 0 0
e− τ 1 τ 2 ( W m ) 1 , : (cid:3) .
Under such choices, (33) is proved and all parameters in (cid:99) W m , (cid:98) B m are bounded by τ 1 .
By composing all convolutional blocks, we have
[Equation]
⋆
[Equation]
⋆ ⋆   .
Lastly, the fully connect layer can be set as
[Equation]
Note that the weights in the fully connected layer are bounded by τ 2 (1 ∨ τ − 1 1 ) .
The above construction gives
[Equation]
B. S UPPORTING L EMMAE FOR T HEOREM
Before stating Lemma 5, we provide a brief definition of cardinal B-splines.
Definition 5 (Cardinal B-spline) .
Let ψ ( x ) = 1 [0 , 1] ( x ) be the indicator function for membership in [0 , 1] .
The cardinal B-spline of order m is defined by taking m + 1 -times convolution of ψ :
[Equation]
where f ∗ g ( x ) ≡ (cid:82) f ( x − t ) g ( t ) dt .
Note that ψ m is a piecewise polynomial with degree m and support [0 , m + 1] .
It can be expressed as [45]
[Equation]
For any k, j ∈ N , let (cid:101) g k,j,m ( x ) =
ψ m (2 k x − j ) , which is the rescaled and shifted cardinal Bspline with resolution 2 − k and support 2 − k
[ j, j + ( m + 1)] .
For k = ( k 1 , . . .
, k d ) ∈ N d and j = ( j 1 , . .
. , j d ) ∈ N d , we define the d dimensional cardinal B-spline as (cid:101) g d k , j ,m ( x ) =
(cid:81) di
=1 ψ m (2 k i x i − j i ) .
When k 1 = . . .
=
k d = k ∈ N , we denote (cid:101) g dk, j ,m ( x ) =
(cid:81) di
=1 ψ m (2 k x
i − j i ) .
B.4. A PPROXIMATING B ESOV FUNCTIONS WITH C ARDINAL B-S PLINES
For any m ∈ N , let J ( k ) =
{− m, − m + 1 , . . .
, 2 k − 1 , 2 k } d and the quasi-norm of the coefficient { c k,j } for k ∈ N , j ∈ J ( k ) be
[Equation]
We can state the following lemma, from DeVore & Popov [13], Dung [17], which provides an upper bound on the error of using cardinal B-splines to approximate functions in B αp,q ([0 , 1] d ) .
Lemma 5 (Lemma 2 in Suzuki [60]; DeVore & Popov [13], Dung [17]) .
Assume that 0 < p, q, r ≤ ∞ and 0 < α < ∞ satisfying α > d (1 /p − 1 /r )
+ .
Let m ∈ N be the order of the cardinal B-spline basis such that 0 < α < min( m, m − 1+1 /p ) .
For any f ∈ B αp,q ([0 , 1] d ) , there exists f N satisfying
[Equation]
for some constant C with N ≫ 1 .
f is in the form of
[Equation]
where { j i } n k i =1 ⊂ J ( k ) , H =
⌈ c 1 log( N ) /d
⌉ , H ∗ =
⌈ ν − 1 log( λN ) ⌉
+ H +1 , n k = ⌈ λN 2 − ν
( k − H )
⌉ for k = H + 1 , . . .
, H ∗ , u
= d (1 /p − 1 /r ) + and ν = ( α − u ) / (2 u ) .
The real numbers c 1 > 0 and λ > 0 are two absolute constants chosen to satisfy (cid:80)
Hk
=1 (2 k + m ) d + (cid:80) H ∗ k
= H +1 n k ≤ N , which are to N .
Moreover, we can choose the coefficients { c k, j } such that
[Equation]
for some constant C 1 .
B.4. A PPROXIMATING C ARDINAL B-S PLINES AND O THERS WITH S INGLE -B LOCK CNN S
The following Proposition 1 quantifies the tradeoff between the size of a single-block CNN and its approximation error for the multiplication operator.
Proposition 1.
Let × be defined as in (13).
For any η ∈ (0 , 1) , there exists a single-block CNN (cid:98) × ( · , · ) such that
[Equation]
where a, b are functions uniformly bounded by c 0 .
(cid:98) × is a single-block CNN approximation of × and is in F SCNN ( L, J, I, τ, τ ) with L
= O (log 1 /η )
+
D layers, J = 24 channels and any 2 ≤ I ≤ D .
All parameters are bounded by τ = ( c 20 ∨ 1) .
Furthermore, the weight matrix in the fully connected layer of (cid:98) × has nonzero entries only in the first row.
Proof of Proposition 1.
First, let us define a particular class of feed-forward ReLU networks of the form
[Equation]
as
F
[Equation]
L
[Equation]
{ f | f
[Equation]
x ) in the form
[Equation]
36) with L layers and width at most J ,
[Equation]
By Proposition 3 in Yarotsky, there exists a feed-forward ReLU network that can approximate the multiplication operation between values with magnitude bounded by c 0 , with η error.
Such feedforward network has O (log 1 /η ) layers, whose width is all bounded by 6 , and all its parameters are bounded by c 2 0 .
Therefore, such a feed-forward network is sufficient to approximate × with η error in L ∞ -norm, because the arguments of × are uniformly bounded c 0 by Assumption 2.
Furthermore, by Lemma 8 in Liu et al.
[42], we can express the aforementioned feed-forward network with a single-block CNN in F SCNN ( L, J, I, τ, τ ) , where L, J, I, τ are as specified in the statement of the proposition.
Proposition 2 quantifies the tradeoff between the size of a single-block CNN and its approximation error for the cardinal B-spline f i ◦ ϕ − 1 i .
Proposition 2 (Proposition 3 in Liu et al.
Let f i ◦ ϕ − 1 i be defined as in (13).
For any δ ∈ (0 , 1) , set N = C
δ − d/α .
For any 2 ≤ I ≤ D , there exists a set of single-block CNNs (cid:110) (cid:98)
f SCNN (cid:111) N j =1 such that
[Equation]
where C 1 is a constant depending on α, p, q and d .
(cid:98) f
SCNN i,j is a single-block CNN approximation of (cid:101)
f i,j (defined in (14)) in F SCNN ( L, J, I, τ, τ ) with
L = O
(log(1 /δ )) , J =
[Equation] [Equation]
[Equation]
The constant hidden in O ( · ) depends on d, α, 2 d αp − d , p, q, c 0 .
Proposition 3 quantifies the tradeoff between the size of the sub-networks for the chart determination network and its approximation error for the chart determination indicators and the distance function d 2 i .
Proposition 3 (Lemma 9 in Liu et al.
Let d 2 i and 1 [0 ,β 2 ] be defined as in (18).
For any θ ∈ (0 , 1) and ∆ ≥ 8 B 2 Dθ , there exists a single-block CNN (cid:98) d 2 i approximating d 2 i such that
[Equation]
and a CNN (cid:98) 1 ∆ approximating 1 [0 ,β 2 ] with
[Equation]
1 ∆ ( x ) =    1 , if a ≤ (1 − 2 − k )( β 2 − 4 B 2 Dθ ) , 0 , if a ≥ β 2 − 4 B 2 Dθ, 2 k (( β 2 − 4 B 2 Dθ ) − 1 a − 1) , otherwise .
for x ∈ X .
The single-block CNN for (cid:98) d 2 i has O (log(1 /θ )) layers, 6 D channels and all weights parameters are bounded by 4 B 2 .
The single-block CNN for (cid:101) 1 ∆ has (cid:6) log( β 2 / ∆) (cid:7) layers, 2 channels.
All weight parameters are bounded by max(2 , | β 2 − 4 B 2 Dθ | ) .
As a result, for any x ∈ X , (cid:98) 1 ∆ ◦ (cid:98) d 2 i ( x ) gives an approximation of 1 U i satisfying (cid:98) 1 ∆ ◦ (cid:98) d 2 i ( x ) =
  1 , if x ∈ U
i and d 2 i ( x ) ≤ β 2 − ∆; 0 , if x / ∈ U i ; between 0 and 1 , otherwise .
B.4. L EMMAE ABOUT S UMMATION AND C OMPOSITION OF CNN
Lemma 6 states that the composition of two single-block CNNs can be expressed as one single-block
CNN with augmented architecture.
Lemma 6.
Let F SCNN1 ( L 1 , J 1 , I 1 , τ 1 , τ 1 ) be a CNN architecture from R D →
R and
F SCNN2 ( L 2 , J 2 , I 2 , τ 2 , τ 2 ) be a CNN architecture from R → R .
Assume the weight matrix in the fully connected layer of F SCNN1 ( L 1 , J 1 , I 1 , τ 1 , τ 1 ) and F SCNN2 ( L 2 , J 2 , I 2 , τ 2 , τ 2 ) has nonzero entries only in the first row.
Then there exists a CNN architecture F SCNN ( L, J, I, τ, τ ) from R D
→ R with
L = L 1 + L 2 , J = max( J 1 , J 2 ) , I = max( I 1 , I 2 ) , τ = max( τ 1 , τ 2 ) such that for any f 1 ∈ F SCNN ( L 1 , J 1 , I 1 , τ 1 , τ 1 ) and f 2 ∈ F SCNN ( L 2 , J 2 , I 2 , τ 2 , τ 2 ) , there exists f ∈ F SCNN ( L, J, I, τ, τ ) such that f ( x ) =
f 2 ◦ f 1 ( x ) .
Furthermore, the weight matrix in the fully connected layer of F SCNN ( L, J, I, τ, τ ) has nonzero entries only in the first row.
Lemma 7 states that the sum of n 0 single-block CNNs with the same architecture can be expressed as the sum of n 1 single-block CNNs with modified width.
Lemma 7 (Lemma 7 in Liu et al.
Let { f i } n
i
=1 be a set of single-block CNNs with architecture
F SCNN ( L 0 , J 0 , I 0 , τ 0 , τ 0 ) .
For any integers 1 ≤ n ≤ n 0 and (cid:101) J satisfying n (cid:101) J
= O ( n 0 J 0 ) and (cid:101) J ≥ J 0 , there exists an architecture F SCNN ( L, J, I, τ, τ ) that gives a set of single-block CNNs { g i } ni =1 such that
[Equation]
Such an architecture has
[Equation]
Furthermore, the fully connected layer of f has nonzero elements only in the first row.
C P ROOF OF CNN C LASS C OVERING N UMBER
In this section, we prove a bound on the covering number of the convolutional neural network class used in Algorithm 1.
Lemma 8.
Given δ > 0 , the δ -covering number of the neural network class F ( M, L, J, I, τ 1 , τ 2 , V ) satisfies
[Equation]
τ
[Equation]
(38) where Λ
= ( M
[Equation]
∨
[Equation] [Equation]
[Equation] [Equation]
=
[Equation]
+with (cid:101) ρ = ρ M , (cid:101) ρ + = 1 + MLρ + , ρ = ( JIτ 1 ) L and ρ + =
(1 ∨ JIτ 1 ) L .
With a network architecture as stated in Theorem 1, we have
[Equation]
where O ( · ) hides constant depending on d , α , 2 d αp − d , p , q , c 0 , B , ω and the surface area of X .
C. S UPPORTING L EMMAE AND P ROOFS
Proposition 4 below provides an upper bound on the L ∞ -norm of a series of convolutional neural network blocks in terms of its architecture parameters, e.g. number of layers, number of channels, etc.
Let J ( i ) m be the number of channels in i -th layer of the m -th block, and let I ( i ) m be the filter size of i -th layer in the m -th block.
Q [ i,j ] is defined as
[Equation]
Proposition 4.
For m = 1 , 2 , · · · , M and x ∈ [ − 1 , 1] D , we have
[Equation]
Proof.
[Equation]
(1 ∨ J ( i − 1) m
I ( i ) m τ 1 ) ≤
∥ P ( x ) ∥ ∞ m (cid:89) j =
L j (cid:89) i =
J ( i − 1)
j
I ( i ) j τ 1 + τm (cid:88) k
=
L k
L k (cid:89) i = (1 ∨ J ( i − 1) k
I ( i ) k τ 1 ) m (cid:89) l = j +
L l (cid:89) i =
J ( i − 1) l
I ( i ) l
τ≤ ∥ x ∥ ∞ m (cid:89) j =
L j (cid:89) i =
J ( i − 1)
j
I ( i ) j τ 1 + τm (cid:88) k
=
L k
L k (cid:89) i = (1 ∨ J ( i − 1) k
I ( i ) k τ 1 ) m (cid:89) l = j +
L l (cid:89) i =
J ( i − 1) l
I ( i ) l τ≤ (1 ∨ τ 1 )   m (cid:89) j =
L j (cid:89) i =
J ( i − 1)
j
I ( i ) j τ  (cid:32) 1 + m (cid:88) k
=
L k
L k (cid:89) i = (1 ∨ J ( i − 1) k
I ( i ) k τ 1 ) (cid:33) , where the first two inequalities are obtained by applying Proposition 9 from Oono & Suzuki [54] recursively.
Lemma 9 quantifies the sensitivity of a CNN with respect to small changes in its weight parameters.
This will be used to create a discrete covering for the CNN class.
Lemma 9.
For f, f ′ ∈ F ( M, L, J, I, τ 1 , τ 2 , V ) such that for ϵ > 0 , ∥ W − W ′
∥ ∞ ≤ ϵ , ∥ b − b ′ ∥ ∞ ≤ ϵ , (cid:13) (cid:13)(cid:13) W ( l ) m − W
( l ) m ′ (cid:13) (cid:13)(cid:13) ∞ ≤ ϵ and (cid:13) (cid:13)(cid:13) B ( l ) m − B
( l ) m ′ (cid:13) (cid:13)(cid:13) ∞
≤ ϵ for all m and l , where ( W, b, {{ ( W ( l ) m , B ( l ) m ) }
L m l =1 } Mm =1 ) and ( W ′ , b ′ , {{ ( W ( l ) m ′ , B ( l ) m ′ )
} L m l =1 } Mm =1 ) are the parameters of f and f ′ respectively, we have
[Equation]
where Λ 1 is defined in Lemma 8.
Proof.
For any x ∈ [ − 1 , 1] D , | f ( x ) − f ′ ( x ) | = | W ⊗ Q ( x ) + b − W ′ ⊗ Q ′ ( x ) − b ′
| = | ( W − W ′ )
⊗ Q ( x ) + b − b ′ + W ′ ⊗ ( Q ( x ) − Q ′ ( x ))
| = | ( W − W ′ )
⊗ Q ( x ) + b − b ′ + W ′ ⊗ ( Q ( x ) − Conv W M , B M ( Q ′ ( x ))
+ Conv W M , B M ( Q ′ ( x )) − Q ′ ( x ))
| = (cid:12)(cid:12)(cid:12)(cid:12) (cid:12) ( W − W ′ ) ⊗ Q ( x ) + b − b ′ +
M (cid:88) m =
[Equation] [Equation]
(cid:12)
[Equation] [Equation]
k =
L k L k
[Equation]
i =
[Equation] [Equation]
i
[Equation]
1) k
I
[Equation]
i )
[Equation] [Equation]
ϵ, where (a) is obtained through the following reasoning.
The first term in (a) can be bounded as
[Equation]
where the first inequality uses Proposition 8 from Oono & Suzuki [54] and the last inequality is obtained by invoking Proposition
[Table] [Table]
second term in
[Table]
), it is true that for any m = 1 , · · · , M , we have
[Table]
j τ  (cid:32)
L m (cid:89) i =
J ( i − 1) m
I ( i ) m τ(cid:33) (1 ∨ τ 1 )   m (cid:89) j =
L j (cid:89) i =
J ( i − 1)
j
I ( i ) j τ  (cid:32) 1 + m (cid:88) k
=
L k
L k (cid:89) i = (1 ∨ J ( i − 1) k
I ( i ) k τ 1 ) (cid:33) ϵ ≤ JDτ 
M (cid:89) j =
L j (cid:89) i =
J ( i − 1)
j
I ( i ) j τ  (1 ∨ τ 1 ) (cid:32) 1 +
M (cid:88) k
=
L k
L k (cid:89) i = (1 ∨ J ( i − 1) k
I ( i ) k τ 1 ) (cid:33) ϵ, where (b) is by Proposition 7 from Oono & Suzuki [54], (c) is by Proposition 2 and 4 from Oono &
Suzuki [54], (d) is by Proposition 2 and 5 from Oono & Suzuki [54], and (e) is obtained by invoking
Proposition 4.
C. P ROOF OF L EMMA
Proof of Lemma 8.
We grid the range of each parameter into subsets with width Λ − 1 1 δ , so there are at most 2( τ 1 ∨ τ 2 )Λ 1 δ − 1 different subsets for each parameter.
In total, there are (cid:0) 2( τ 1 ∨ τ 2 )Λ 1 δ − 1 (cid:1)
Λbins in the grid.
For any f, f ′ ∈ F ( M, L, J, I, τ 1 , τ 2 , V ) within the same grid, by Lemma 9, we have ∥ f − f ′
∥ ∞ ≤ δ .
We can construct the ϵ -covering with cardinality (cid:0) 2( τ 1 ∨ τ 2 )Λ 1 δ − 1 (cid:1) Λ 2 by selecting one neural network from each bin in the grid.
Taking log and plugging in the network architecture parameters in Lemma 1, we have log N ( δ, F ( M, L, J, I, τ 1 , τ 2 , V ) , ∥·∥ ∞ )
=
O (cid:0) Λ 2 log (cid:0) ( τ 1 ∨ τ 2 ) Λ 1 δ − 1 (cid:1)(cid:1)
[Equation]
where the inequality is due to Λ
= O ( (cid:102) MDD 2 (cid:101)
J 2 log( (cid:102) M (cid:101) J )) .
By plugging in the choice of τ 1 , ρ =
L M − 1 ≤ M − 1 , so (cid:101) ρ
= (1 + M − 1 ) M ≤ e .
Moreover, (cid:101) ρ + = 1 + ML .
D S TATISTICAL R ESULT OF CNN-B ESOV A PPROXIMATION (L EMMA 10)
In this section, we derive the statistical estimation error for using a CNN empirical MSE minimizer to estimate a Besov ground truth function over an i.i.d.
dataset.
We need to choose the appropriate
CNN architecture and size in order to balance the approximation error from Theorem 1 and variance.
Thsi statistical estimation error can be decomposed into the error of using CNN to approximate
Besov function (Theorem 1), terms that grow with the covering number of our CNN class, and the error of using the discrete covering to approximate our CNN class.
In Theorem 2, we expand the estimation error (cid:98) v π − v π over time steps and upper-bound the amount of estimation error in each time step with Lemma 10.
Details of Theorem 2 are in Appendix A.
Lemma 10.
Let X be a d -dimensional compact Riemannian manifold that satisfies Assumption 1.
We are given a function f 0 ∈ B αp,q ( X ) , where s, p, q satisfies Assumption 2.
We are also given samples S n = { ( x i , y i ) } ni =1 , where x i are i.i.d.
sampled from a distribution P x on X and y i
=
f 0 ( x i ) + ζ i .
ζ i ’s are i.i.d.
sub-Gaussian random noise with variance σ 2 , uncorrelated with x i ’s. If we compute an estimator
[Equation]
with the neural network class F = F ( M, L, J, I, τ 1 , τ 2 , V ) such that
[Equation]
with any integer I ∈ [2 , D ] and (cid:102) M, (cid:101) J > 0 satisfying (cid:102) M (cid:101) J
= O ( n
[Equation] [Equation]
+2 d ) , then we have
[Equation]
where V F = ∥ f 0 ∥ ∞ and the expectation is taken over the training sample S n , and c is a constant depending on D
α
α +2 d , d , α , 2 d αp − d , p , q , c 0 , B , ω and the surface area of X .
O ( · ) hides constant depending on d , α , 2 d αp − d , p , q , c 0 , B , ω and the surface area of X .
First, note that the nonparametric regression error can be decomposed into two terms:
[Equation]
where T 1 reflects the squared bias of using neural networks to approximate ground truth f 0 , which is related to Theorem 1, and T 2 is the variance term.
D. S UPPORTING L EMMAE
Lemma 11 (Lemma 5 in Chen et al.
Fix the neural network class F ( M, L, J, I, τ 1 , τ 2 , V ) .
For any constant δ ∈ (0 , 2 V ) , we have
[Equation] [Equation]
where
N ( δ, F ( M, L, J, I, τ 1 , τ 2 , V ) , ∥·∥ ∞ ) denotes the δ -covering number of
F ( M, L, J, I, τ 1 , τ 2 , V ) with respect to the ℓ ∞ norm, i.e., there exists a discretization of
F ( M, L, J, I, τ 1 , τ 2 , V ) into N ( δ, F ( M, L, J, I, τ 1 , τ 2 , V ) , ∥·∥ ∞ ) distinct elements, such that for any f ∈ F , there is ¯ f in the discretization satisfying (cid:13)(cid:13)
¯ f − f
(cid:13)(cid:13) ∞ ≤ ϵ .
Lemma 12 (Lemma 6 in Chen et al.
For any constant δ ∈ (0 , 2 R ) , T 2 satisfies
[Equation]
D. P ROOF OF L EMMA
Proof of Lemma 10.
Recall that the bias and variance decomposition of
E (cid:20) (cid:82) X (cid:16) (cid:98) f n ( x ) − f 0 ( x ) (cid:17)
2 d P x ( x ) (cid:21) as
E (cid:20)(cid:90)
X (cid:16) (cid:98) f n ( x ) − f 0 ( x ) (cid:17) 2 d P x
[Equation]
Applying the upper bounds of T 1 and T 2 in Lemmas 11 and 12 respectively
[Equation]
we can derive
E (cid:20)(cid:90)
[Equation]
We need there to exist a network in F ( M, L, J, I, τ 1 , τ 2 , V ) which can yield a function f satisfying ∥ f − f 0 ∥ ∞ ≤ ϵ for ϵ ∈ (0 , 1) .
ϵ will be chosen later to balance the bias-variance tradeoff.
In order to achieve such ϵ -error, we set (cid:102) M (cid:101)
J
= ϵ − d/α , so we now have our network architecture as specified in Theorem 1 in terms of ϵ .
Then, we can use the parameters in this architecture to invoke the upper bound of the covering number in Lemma 8:
[Equation]
where O ( · ) hides constant depending on log D , d , α , 2 d αp − d , p , q , c 0 , B , ω and the surface area of X .
Plugging it in, we have
[Equation] [Equation]
Finally we choose ϵ to satisfy ϵ 2 = 1 n D 3 ϵ − dα , which gives ϵ = D
α 2 α + d n − α 2 α + d .
It suffices to pick δ = 1 n .
Substituting both ϵ and δ into (41), we deduce the desired estimation error bound
[Equation]
where constant c depends on D
α
α + d , d , α , 2 d αp − d , p , q , c 0 , B , ω and the surface area of X .
E A R ESULT FOR F EED -F ORWARD R E LU N EURAL N ETWORK E. F EED -F ORWARD R E LU N EURAL N ETWORK
We consider multi-layer ReLU (Rectified Linear Unit) neural networks [25].
ReLU activation is popular in computer vision, natural language processing, etc.
because the vanishing gradient issue is less severe with it, which is nonetheless common with its counterparts like sigmoid or hyperbolic tangent activation [25, 27].
An L -layer ReLU neural network can be expressed as
[Equation]
in which W 1 , · · · , W L and b 1 , · · · , b L are weight matrices and vectors and ReLU( · ) is the entrywise rectified linear unit, i.e. ReLU( a ) =
max { 0 , a } .
The width of a neural network is defined as the number of neurons in its widest layer.
For notational simplicity, we define a class of neural networks
F ( L, p, I, τ, V ) =
{ f | f ( x ) in the form (42) with L layers and width at most p ,
[Equation]
E. P OLICY E VALUATION E RROR AND I TS P ROOF
From this point, we denote the function class F ( L, p, I, τ, V ) , whose parameters L, p, I, τ, V are chosen according to Theorem 3, with the shorthand F .
In this section, this F is used in Algorithm 1, instead of the CNN class in (11).
Theorem 3.
Suppose Assumption 1 and 2 hold.
By choosing
[Equation]
in Algorithm 1, in which O ( · ) hides factors depending on α , d and log D , we have
[Equation]
[Equation]
in which the expectation is taken over the data, and C is a constant depending on log D , α , B , d , ω , the surface area of X and c 0 .
The distributional mismatch is captured by
[Equation]
in which Q is the Minkowski sum between the ReLU function class and the Besov function class, i.e., Q = { f + g | f ∈ B αp,q ( X ) , g ∈ F} .
Proof of Theorem 3.
The goal is to bound
[Equation]
To get an expression for that, we first expand it recursively.
To illustrate the recursive relation
[Equation]
we examine the quantity at step h :
[Equation]
[Table]
[Equation]
where C denotes a (varying) constant depending on log D , α , B , d , ω , the surface area of X and c 0 .
In (a), note T πh (cid:98) Q πh +1 ∈ B αp,q ( X ) by Assumption 2 and − (cid:98) T πh (cid:16) (cid:98) Q πh +1 (cid:17) ∈ F by our algorithm, so
T πh (cid:98) Q πh +1 − (cid:98) T πh (cid:16) (cid:98) Q πh +1 (cid:17) ∈ Q .
Then we obtain this inequality by invoking the following lemma.
In (b), we use Jensen’s inequality and the fact that square root is concave.
To obtain (c), we invoke the following lemma, which provides an upper bound on the regression error.
Specifically, we will use Lemma 13 when conditioning on D h +1 , · · · , D H , i.e. the data from time step h
+ 1 to time step H .
Note that after conditioning, T πh (cid:98) Q πh +1 becomes measurable and deterministic with respect to D h +1 , · · · , D H .
Also, D h +1 , · · · , D H are independent from D h , which we use in the regression at step h .
To justify our use of Lemma 13, we need to cast our problem into a regression problem described in the lemma.
Since { ( s h,k , a h,k ) } Kk =1 are i.i.d.
from q π 0 h , we can view them as the samples x i ’s in the lemma.
We can view T π h (cid:98) Q π h +1 , which is measurable under our conditioning, as f 0 in the lemma.
Furthermore, we let
[Equation]
In order to invoke Lemma 13 under the conditioning on D h +1 , · · · , D H , we need to verify whether three conditions are satisfied (conditioning on D h +1 , · · · , D H ):
Sample { ( s h,k , a h,k ) }
Kk =1 are i.i.d; 2.
Sample { ( s h,k , a h,k ) }
Kk =1 and noise { ζ h,k } Kk =1 are uncorrelated; 3.
Noise { ζ h,k } Kk =1 are independent, zero-mean, subgaussian random variables.
In our setting, { ( s h,k , a h,k ) } Kk =1 are i.i.d.
from q π 0 h .
Due to the time-inhomogeneous setting, they are independent from D
h
+1 , · · · , D H , so { ( s h,k , a h,k ) }
Kk =1 are still i.i.d.
under our conditioning.
Thus, Condition 1 is clearly satisfied.
We may observe that under our conditioning, the transition from ( s h,k , a h,k ) to s ′ h,k is the only source of randomness in ζ h,k , besides ( s h,k , a h,k ) itself.
The distribution of ( s h,k , a h,k , s ′ h,k ) is actually the product distribution between P h ( ·| s h,k , a h,k ) and q π 0 h , so a function of s
′ h,k , generated from the transition distribution P h ( ·| s h,k , a h,k ) , is uncorrelated with ( s h,k , a h,k ) .
Thus, ( s h,k , a h,k ) ’s are uncorrelated with ζ h,k ’s under our conditioning, and Condition 2 is satisfied.
Condition 3 can also be easily verified.
Under our conditioning, the randomness in ζ h,k only comes from ( s h,k , a h,k , s ′ h,k , r h,k ) , which are independent from ( s h,k ′ , a h,k ′ , s ′ h,k ′ , r h,k ′ ) for any k ′ ̸
=
k , so ζ h,k ’s are independent from each other.
As for the mean of ζ h
[Equation]
=
+ 0 = 0 .
On the other hand, (cid:13)(cid:13)(cid:13) (cid:98) Q πh +1 (cid:13)(cid:13)(cid:13) ∞ ≤ H almost surely, because it is a function in our ReLU network class
F .
Thus, ζ h,k is a bounded random variable with ζ h,k ∈ [ − 2 H, 2 H ] almost surely, so its variance is bounded by 4 H 2 .
Its boundedness also implies it is a subgaussian random variable.
Thus, Condition 3 is also satisfied.
Hence, Lemma 13 proves, for step h in our algorithm,
[Equation]
Note that this upper bound holds for any (cid:98) Q πh +1 or D h
+1 , · · · , D H .
The sole purpose of our conditioning is that we could view (cid:98) Q πh +1 as a measurable or deterministic function under the conditioning and then apply Lemma 13.
Therefore,
[Equation]
Finally
[Equation]
we carry out the recursion from time step 1 to time step H
[Equation]
and the final result is
[Equation]
E. L EMMA 13 AND I TS P ROOF
Lemma 13.
Let X be a d -dimensional compact Riemannian manifold isometrically embedded in R D with reach ω .
There exists a constant B > 0 such that for any x ∈ X , | x j | ≤ B for all j = 1 , · · · , D .
We are given a function f 0 ∈ B αp,q ( X ) and samples S n = { ( x i , y i ) } ni =1 , where x i are i.i.d.
sampled from a distribution P x on X and y i
=
f 0 ( x i ) + ζ i .
ζ i ’s are i.i.d.
sub-Gaussian random noise with variance σ 2 , uncorrelated with x i ’s. If we compute an estimator
[Equation]
with the neural network class F = F ( L, p, I, τ, V ) such that
[Equation]
) then we have
[Equation]
where V F = ∥ f 0 ∥ ∞ and the expectation is taken over the training sample S n , and c is a constant depending on log D , α , B , d , ω , the surface area of X and c 0 .
Proof of Lemma 13.
Recall that the bias and variance decomposition of
[Equation]
Applying the upper bounds of T 1 and T 2 in Lemmas 11 and 12 respectively
[Equation]
we can derive
[Equation]
We need there to exist a network in F ( L, p, I, τ, V ) which can yield a function f satisfying ∥
f − f 0 ∥ ∞ ≤ ϵ for ϵ ∈ (0 , 1) .
ϵ will be chosen later to balance the bias-variance tradeoff.
By
Lemma 2 of Nguyen-Tang et al.
[51], in order to achieve such ϵ -error, we need
[Equation]
where O ( · ) hides factors of log D , α , d and the surface area of X , so we now have our network architecture as specified in Theorem 1 in terms of ϵ .
Then, we can use the architecture parameters in (13) to invoke the upper bound of the covering number in Lemma 7 of Chen et al.
[Equation] [Equation]
where c ′′ is a constant depending on log B , ω and log log n .
Plugging it in, we have
[Equation]
Finally we choose ϵ to satisfy ϵ 2 = 1 n ϵ − dα , which gives ϵ = n − α 2 α + d .
It suffices to pick δ = 1 n .
Substituting both ϵ and δ into (48), we deduce the desired estimation error bound
[Equation]
where constant c depends on log D , d , α , 2 d αp − d , p , q , c 0 , B , ω and the surface area of X .
F S UPPLEMENT FOR E XPERIMENTS
F.
D ETAILS
FOR E XPERIMENTS WITH C ART P OLE
We use the CartPole environment from OpenAI gym.
We consider it as a time-inhomogeneous finite-horizon MDP by setting a time limit of 100 steps.
We turn the terminal states in the original
CartPole into absorbing states, so if a trajectory terminates before 100 steps, the agent would keep receiving zero reward in its terminal state until the end.
The target policy is a policy trained foriterations using REINFORCE, in which each iteration samples for 100 trajectories with truncation after 150 time steps.
The target policy value v π is estimated to be 65 . 2117 , which we obtain by
Monte Carlo rollout from the initial state distribution.
For a given behavior policy, to obtain dataset D h at time step h , we sample for K independent episodes under the behavior policy and only take the ( s, a, s ′ , r ) tuple from the h -th transition in each episode.
This is an excessive way to guarantee the independence among these K samples; in practice, we could directly sample from a sampling distribution.
We sample for D h for each h = 1 , · · · , 100 .
We use the render function in OpenAI gym for the visual display of CartPole.
We downsample images to the desired resolution via cubic interpolation.
A high-resolution image (see Figure 3) is represented as a 3 × 40 × 150 RGB array; a low-resolution image (see Figure 4) is represented as a 3 × 20 × 75 RGB array.
For the function approximator in FQE, we use a neural network that comprises 3 convolutional layers each with output channel size 16 , 32 and 32 and a final linear layer.
These layers are interleaved with
ReLU activation and batch norm layers for weight normalization.
For high resolution input, we use kernel size 5 and stride 2 ; for low resolution input, we use kernel size 3 and stride 1 .
For experiments with high resolution, in each step of FQE, we solve the regression by training the network via stochastic gradient descent with batch size 256 for 20 epochs.
In high-resolution experiments, we use 0 .
01 learning rate; in low-resolution experiments, we use 0 .
001 learning rate.
We compute the average and standard deviation of FQE’s result over 5 random seeds.
F. E XPERIMENTS WITH L UNAR L ANDER
In addition to our CartPole experiments, we present numerical experiments for evaluating FQE with
CNN function approximation on the LunarLander environment [5].
The LunarLander problem has a 8 -dimensional continuous intrinsic state space.
We consider a finite-horizon MDP with horizon H = 100 in this environment.
In our experiments, we solve the OPE problem with FQE (Algorithm 1).
We take the visual display of the environment as states.
These images serve as a high-dimensional representation of LunarLander’s original 8 -dimensional continuous state space.
In our algorithm, we use a deep CNN to approximate the Q-functions and solve the regression with SGD.
The results are report in Table 3.
[Table]
The experiment setup is almost the same as our CartPole experiments.
We conduct this experiment in two cases: (A) data are generated from the target policy itself; (B) data are generated from a mixture policy of 0 .
9 target policy and 0 .
1 uniform distribution.
A high-resolution image (see Figure 5) is represented as a 3 × 40 × 70 RGB array; a low-resolution image (see Figure 6) is represented as a 3 × 20 × 35 RGB array.
[Figure]
[Equation]
[Figure]
ATE IN C ART P OLE
We plot the relative error of FQE in our CartPole experiments from Section 5.
Figure 7 and 8 show that the estimation error follows an exponential decay.
We also plot this relative error in log-log plots (Figure 9 and 10).
We can observe that the curves are linear, which confirms the form of our theoretical bound in Theorem 2.
Moreover, we can observe that the slope, which represents the decay rate of the estimation error, is generally the same between high-resolution and low-resolution experiments.
This confirms our theory that the decay rate takes little influence from the ambient dimension.
[Figure] [Figure]
[Figure] [Figure]