P ROVABLY E FFICIENT N EURAL O FFLINE R EINFORCEMENT L EARNING VIA P ERTURBED R EWARDS
A BSTRACT
We propose a novel offline reinforcement learning (RL) algorithm, namely Value
Iteration with Perturbed Rewards (VIPeR) which amalgamates the randomized value function idea with the pessimism principle.
Most current offline RL algorithms explicitly construct statistical confidence regions to obtain pessimism via lower confidence bounds (LCB), which cannot easily scale to complex problems where a neural network is used to estimate the value functions.
Instead, VIPeR implicitly obtains pessimism by simply perturbing the offline data for multiple times with carefully-designed i.i.d Gaussian noises to learn an ensemble of estimated state-action values and acting greedily to the minimum of the ensemble.
The estimated state-action values are obtained via fitting a parametric model (e.g. neural networks) to the perturbed datasets using gradient descent.
As a result,
VIPeR only needs O (1) time complexity for action selection while LCB-based algorithms require at least Ω( K 2 ) , where K is the total number of trajectories in the offline data.
We also propose a novel data splitting technique that helps remove the potentially large log covering number in the learning bound.
We prove that VIPeR yields a provable uncertainty quantifier with overparameterized neural networks and achieves an ˜ O (cid:16) κH 5 / 2 ˜ d √ K (cid:17) sub-optimality where ˜ d is the effective dimension, H is the horizon length and κ measures the distributional shift.
We corroborate the statistical and computational efficiency of VIPeR with an empirical evaluation in a wide set of synthetic and real-world datasets.
To the best of our knowledge, VIPeR is the first offline RL algorithm that is both provably and computationally efficient in general Markov decision processes (MDPs) with neural network function approximation.
I NTRODUCTION
Offline reinforcement learning (offline RL) (Lange et al., 2012; Levine et al., 2020) is a practical paradigm of RL for domains where exploration is prohibitively expensive or even implausible but a fixed dataset of previous experiences is available a priori.
Offline RL finds vast applications in a number of critical domains including healthcare (Gottesman et al., 2019; Nie et al., 2021), recommendation systems (Strehl et al., 2010; Thomas et al., 2017), econometrics (Kitagawa & Tetenov, 2018; Athey & Wager, 2021), whereas it has also recently attracted substantial research efforts and yielded several important empirical successes (Chen et al., 2021; Wang et al., 2022b;a; Meng et al., 2021), just to name a few.
A core problem in offline RL is to efficiently exploit the offline dataset to learn an optimal policy without any further exploration as in the online RL.
To deal with this exploration-free policy learning problem, dominant approaches incorporate uncertainty from the offline dataset into the decision making (Buckman et al., 2020; Jin et al., 2021; Xiao et al., 2021;
Nguyen-Tang et al., 2022a;
Ghasemipour et al., 2022; Brandfonbrener et al., 2022; An et al., 2021; Bai et al., 2022).
The main idea for the uncertainty-aware approach for offline RL is the pessimism principle which constrains the learned policy to the offline data and leads to various lower confidence bound (LCB) algorithms.
However, these methods cannot easily extend or scale to complex problems where neural function approximation is used to estimate the value functions.
In particular, it is costly to explicitly compute the statistical confidence regions of the model or value functions if the function approximator is overparameterized neural networks.
For example, the LCB construction for neural offline contextual bandits (Nguyen-Tang et al., 2022a) and RL (Xu & Liang, 2022) requires the inverse of a large covariance matrix of the size as large as the number of parameters in the neural network.
This computational cost hinders the practical application of these provably efficient offline RL algorithms.
Thus, a largely open problem is the design of a computationally and provably efficient offline RL algorithm with neural network function approximation .
In this work, we address the problem above by combining the randomized value function idea (Osband et al., 2016; Ishfaq et al., 2021) and the pessimism principle (Jin et al., 2021).
The algorithm is strikingly simple: we randomly perturb the offline rewards several times and act greedily with respect to the minimum of the estimated state-action values.
The intuition is that taking the minimum from an ensemble of randomized state-action values can efficiently achieve pessimism with high probability while avoiding explicitly computing any statistical confidence regions.
We consider optimization in our algorithm where the estimated state-action values are obtained via training of the neural network with gradient descent (GD).
Moreover, we propose a novel data splitting technique that helps remove the dependence on the potentially large log covering number in the learning bound.
We show that our proposed algorithm yields a provable uncertainty quantifier with overparameterized neural network function approximation and obtains an ˜ O (cid:16) κH 5 / 2 ˜ d √ K (cid:17) sub-optimality where K is the total number of episodes in the offline data, ˜ d is the effective dimension, H is the horizon length and κ measures the distributional shift.
The provable efficiency is achieved along with computational efficiency as our algorithm needs only O (1) time complexity for action selection while LCB-based algorithms require O ( K 2 ) time complexity.
We empirically corroborate the statistical and computational efficiency of our proposed algorithm in a wide set of synthetic and realworld datasets.
The experimental results show that our proposed algorithm has a strong advantage in computational efficiency while even outperforming LCB-based neural algorithms.
To our knowledge, our proposed method is the first offline RL algorithm that is both provably and computationally efficient in general MDPs with neural network function approximation.
R ELATED W ORK
Randomized value functions for RL.
For online RL, Osband et al.
(2016; 2019) were the first to propose randomized value functions for exploration where the learner plans with respect to randomized value function estimates.
This idea was inspired by posterior sampling for RL (Osband et al., 2013) that samples a value function from a posterior distribution and acts greedily with respect to the sampled function.
This algorithm generates randomized value functions by injecting Gaussian noise into the training data and fit a model into the perturbed data.
Jia et al.
(2022) extended the perturbed reward idea to online contextual bandit with neural function approximation.
Ishfaq et al.
(2021) obtained provably efficient online RL with general function approximation using the perturbed rewards.
While the randomized value functions are intuitive to obtain optimism for online
RL, for offline RL, under distributional shift, it can be paradoxically difficult to properly obtain pessimism with randomization and ensemble (Ghasemipour et al., 2022), especially with neural function approximation.
In our work, we propose a simple trick that generates multiple randomized value functions and takes the minimum to enforce sufficient pessimism for offline RL that is both provably and computationally efficient.
Offline RL with function approximation.
Provably efficient offline RL has been extensively studied under linear function approximation.
Jin et al.
(2021) were the first to show that pessimistic value iteration is provably efficient for offline linear MDPs.
Xiong et al.
; Yin et al.
(2022a) improved Jin et al.
(2021) by leveraging variance reduction.
Xie et al.
(2021) proposed Bellmanconsistent assumption with general function approximation which improves the bound of Jin et al.
(2021) by a factor of √ d when realized to finite action space and linear MDPs.
Wang et al.
(2020a);
Zanette (2021) studied the statistical hardness of offline RL with linear function approximation via exponential lower bound and Foster et al.
(2021) suggested that only realizability and strong uniform data coverage are not sufficient for sample-efficient offline RL.
Beyond linearity, some works study offline RL in general, nonparametric and parametric function approximation, either based on Fitted- Q Iteration (FQI) (Munos & Szepesv´ari, 2008; Le et al., 2019; Chen & Jiang, 2019; Duan et al., 2021a;b; Hu et al., 2021; Nguyen-Tang et al., 2022b; Ji et al., 2022) or pessimism principle (Uehara & Sun, 2021; Nguyen-Tang et al., 2022a; Jin et al., 2021).
While pessimism-based algorithms avoid the strong assumptions of data coverage used by FQI-based algorithms, they require an explicit construction of valid confidence regions and possibly the inverse of large covariance matrix which are computationally prohibited and cannot easily scale to complex function approximation setting.
This limits the applicability of pessimism-based provably efficient offline RL to practical settings.
Recent work Bai et al.
(2022) has proposed to estimate the uncertainty for constructing LCB via the disagreement of bootstrapped Q-functions.
However, the uncertainty quantifier is only guaranteed in linear MDPs and also needs to be explicit.
We also provide more detailed discussion of our technical contribution placing in the context of the literature, in Section C.1.
P RELIMINARIES
In this section, we provide basic background on offline RL and overparameterized neural networks.
3. E PISODIC TIME - INHOMOGENOUS M ARKOV DECISION PROCESSES (MDP S )
A finite-horizon Markov decision process (MDP) is denoted as the tuple M =
( S , A , P , r, H, d 1 ) , where S is an arbitrary state space,
A an arbitrary action space, H the episode length, and d 1 the initial state distribution.
We assume that SA := |S||A| is finite but arbitrarily large, e.g. it can be as large as the total number of atoms in the observable universe ≈ 10 82 .
A time-inhomogeneous transition kernel P = { P h } Hh =1 , where P h : S ×
A → P ( S ) (where P ( S ) denotes the set of probability measures over S ) maps each state-action pair ( s h , a h ) to a probability distribution P h ( ·| s h , a h ) , and r = { r h } Hh =1 where r h : S
×
A → [0 , 1] is the mean reward function at step h .
A policy π = { π h } Hh =1 assigns each state s h ∈ S to a probability distribution, π h ( ·| s h ) , over the action space and induces a random trajectory s 1 , a 1 , r 1 , . . .
, s H , a H , r H , s H
+
where s 1 ∼ d 1 , a h ∼ π h ( ·| s h ) , s h +1 ∼ P h ( ·| s h , a h ) .
We define the state value function V πh ∈ R S and the actionstate value function Q πh ∈ R S×A at each timestep h as Q πh ( s, a ) =
E π [ (cid:80)
Ht
= h r t | s h
= s, a h
= a ] , and V πh ( s ) =
E a ∼ π ( ·| s )
[ Q πh ( s, a )] , where the expectation E π is taken with respect to the randomness of the trajectory induced by π .
For any V : S → R , we define the Bellman operator at timestep h as ( B h V )
( s, a ) :
= r h ( s, a ) + ( P h V )( s, a ) , where P h is the transition operator defined as ( P h V )( s, a ) :=
E s ′ ∼ P h ( ·| s,a )
[ V ( s ′ )] .
The Bellman equations are given as follows.
For any ( s, a, h ) ∈ S ×
A ×
[ H ] ,
[Equation]
where
[ H ] := { 1 , 2 , . . .
, H } , and ⟨· , ·⟩ A denotes the inner product over A .
We define an optimal policy π ∗ as any policy that yields the optimal value function, i.e. V π ∗ h ( s ) =
sup π V π h ( s ) for any ( s, h ) ∈ S ×
[ H ] .
For simplicity, we denote V π ∗ h and Q π ∗ h as V ∗ h and Q ∗ h , respectively.
The Bellman optimality equation can be written as
[Equation]
We define the occupancy density as d πh ( s, a ) :
= P (( s h , a h ) =
( s, a ) | π ) which is the probability that state-action pair at timestep h is ( s, a ) if the policy π is followed.
We write d π ∗ h as d ∗ h .
Offline regime.
In the offline regime, the RL learner is given a fixed dataset D = { ( s t h , a t h , r t h , s t h +1 ) } t ∈ [ K ] h ∈
[ H ] generated a priori by some unknown behaviour policy µ = { µ h } h ∈ [ H ] , where K is the total number of trajectories, a th ∼ µ h ( ·| s th ) , s th +1 ∼ P h ( ·| s th , a th ) for any ( t, h ) ∈
[ K ] ×
[ H ] .
Note that we allow the trajectory at any time t ∈
[ K ] to depend on the trajectories at the previous times.
The goal of offline RL is to learn a policy ˆ π only from D such that it minimizes the sub-optimality of ˆ π , which is defined as SubOpt
[Equation]
ˆ π ) :
=
{+E s 
+}
{+∼ d 
 [SubOpt(ˆ π ; s 1 )]+} , where SubOpt(ˆ π ; s 1 ) : =
V π
[Equation]
1 ( s
[Equation]
V
[Equation]
π 1 ( s 1 ) .
Notation.
For simplicity, we write x th =
( s th , a th ) and x = ( s, a ) .
We write ˜ O ( · ) to hide logarithmic factors of the problem parameters ( d, H, K, m, 1 /δ ) in the standard Big-Oh notation.
We use Ω( · ) as the standard Omega notation.
We write u ≲ v if u = O ( v ) and write u ≳ v if v ≲ u .
We write A ⪯ B iff B − A is a positive definite matrix.
We denote I d the identity matrix in R d .
3. O VERPARAMETERIZED N EURAL N ETWORKS
In this paper, we consider neural function approximation setting where the state-action value function is approximated by a two-layer neural network.
For simplicity, we denote X := S ×A and view it as a subset of R d .
Without loss of generality, we assume
X ⊂ S d − 1 := { x ∈ R d : ∥ x ∥ 2 = 1 } .
We consider a standard two-layer neural network: f ( x ; W, b ) = 1 √ m (cid:80) mi
=1 b i σ ( w Ti x ) , where m is an even number, σ ( · ) = max {· , 0 } is the ReLU activation function (Agarap, 2018), and
W = ( w T 1 , . . .
, w Tm ) T ∈ R md .
During the training, we initialize ( W, b ) via the symmetric initialization scheme (Gao et al., 2019) as follows:
For any i ≤ m 2 , w i = w m
+ i ∼ N (0 , I d /d ) , and b m
+ i = − b i ∼ Unif ( {− 1 , 1 } ) .
1 For simplicity, during training we fix all b
i after initialization and only optimize over W , thus we write f ( x ; W, b ) as f ( x ; W ) .
Denote g ( x ; W ) = ∇ W f
( x ; W ) ∈ R md , and let W 0 be the initial parameters of W .
We assume the neural network is overparameterized in the sense that the width m is sufficiently larger than the number of samples K .
Overparameterization has been shown to be effective to study the convergence and interpolation phenomenon of neural networks (Arora et al., 2019; Allen-Zhu et al., 2019; Hanin & Nica, 2019; Cao & Gu, 2019;
Belkin, 2021).
Under such an overparameterization regime, the dynamics of the training of the neural network can be captured by the framework of neural tangent kernel (NTK) (Jacot et al., 2018).
A LGORITHM
1: Input : Offline data D = { ( s kh , a kh , r kh ) } k ∈
[ K ] h ∈ [ H ] , a parametric function family F = { f ( · , · ; W ) :
W ∈ W} ⊂ {X → R } (e.g. neural networks), perturbed variances { σ h } h ∈ [ H ] , number of bootstraps M , regularization parameter λ , step size η , number of gradient descent steps J , and cutoff margin ψ , split indices {I h } h ∈
[ H ] where I h :=
[( H − h ) K ′ + 1 , . .
, ( H − h + 1) K ′ ] 2: Initialize ˜ V
H +1 ( · ) ← 0 and initialize f ( · , · ; W ) with initial parameter W3:
for h = H, . . .
, 1 do 4: for i = 1 , . . .
, M do 5:
Sample { ξ k,ih } k ∈I h ∼ N (0 , σ 2 h ) and ζ ih = { ζ j,ih } j ∈
[ d ] ∼ N (0 , σ 2 h I d ) 6:
Perturb the dataset ˜ D ih ← { s kh , a kh , r kh + ˜ V h +1 ( s kh +1 )
+ ξ k,ih } k ∈I h 7:
Let ˜ W ih ← GradientDescent ( λ, η, J, ˜ D
ih , ζ ih , W 0 ) (Algorithm 2) 8: end for 9:
Compute ˜ Q h ( · , · ) ← min { min i ∈
[ M ] f ( · , · ; ˜ W i h ) , ( H − h + 1)(1 + ψ ) } + 10: ˜
π h ← arg max π h ⟨ ˜ Q h , π h ⟩ and ˜ V
h ← ⟨ ˜ Q h , ˜ π h ⟩ 11: end for 12: Output : ˜ π
= { ˜ π h } h ∈
[ H ] .
In this section, we propose Value Iteration with Perturbed Rewards (VIPeR) algorithm for offline
[-= { s kh , a kh , r kh + ˜ V h +
 ( s kh +1 )-]RL (Algorithm 1).
[-h +
-]
[-+
-]
[-+ ξ k,i h )) 
-]
The key idea of VIPeR is to train a parametric model (e.g. a neural network) on a perturbed-reward dataset for several times and simply act greedily with respect to the minimum of the estimated state-action values.
In particular, at each timestep h ∈
[ H ] , we perturb the offline rewards with σ h -variance Gaussian noise for M times and combine with the upper-timestep estimated state value ˜
V h +1 to obtain a perturbed dataset { ˜ D i h } i ∈
[ M ] (Line 6).
VIPeR then updates ˜ W i h by minimizing the perturbed regularized squared loss on { ˜ D ih }
i ∈ [ M ] using gradient descent (Line 7).
The returned policy is greedy with respect to the truncated version of the minimum of the ensemble of the estimated state-action values (Line 9 and 10).
We truncate the ensemble minimum by ( H − h + 1)(1 + ψ ) (Line 9) where ψ ≥ 0 is the small cutoff margin that is for the theoretical analysis and will be chosen later.
It is important to note that we split the trajectory indices [ K ] evenly into H disjoint buckets [ K ] = ∪ h ∈
[ H ] I h , where I h
=
[( H − h ) K ′ + 1 , . .
, ( H − h + 1) K ′ ] with K ′ :=
⌊ K/H ⌋ 2 , as illustrated in Figure 1.
The estimated ˜ Q h is thus obtained only from the offline data with (trajectory) indices from I h along with
˜ V h
+1 .
This novel design removes the data dependence structure in offline RL with function approximation (Nguyen-Tang et al., 2022b) and avoids a log covering number term in the sub-optimality bound, as we show in Subsection D.1. 1: Input:
Regularization parameter λ , step size η , number of gradient descent steps
J , perturbed dataset ˜ D
ih
=
{ s kh , a {+kh , r kh + ˜ V h +
+} ( s kh +1 ) + ξ t,ih } k ∈I h , regularization perturber ζ ih , initial parameter W2:
L ( W ) :
= 12 (cid:80)
k ∈I h
( f ( s kh , a kh ; W ) − ( r kh + ˜ V
{+h +
+}
( s kh +1 )
+ ξ k,ih ))
2 + λ 2 ∥ W + ζ ih − W 0 ∥3: for j = 0 , . . .
, J − 1 do 4:
W j +1 ← W j − η ∇ L ( W j ) 5: end for 6: Output: W J .
To deal with non-linearity of the underlying
MDP, we adopt a two-layer fully connected neural network for the parametric function family F in Algorithm 1 to approximate the actionstate values: f ( x ; W ) = 1 √ m (cid:80) mi
=1 b i σ ( w Ti x ) , as defined in Subsection 3.2.
We use twolayer neural networks to simplify the later analysis.
We adopt gradient descent to train for the randomized neural state-action value functions { f ( · , · ; ˜ W ih ) } i ∈
[ M ] .
The use of gradient descent is for analysis convenience and we can extend to stochastic gradient descent based on recent works in overparameterized neural networks (Allen-Zhu et al., 2019; Cao & Gu, 2019) with more involved analysis.
Existing offline RL algorithms explicitly construct statistical confidence regions to perform pessimism for the offline setting.
Such explicit confidence construction is computationally expensive in complex problems where a neural network is used as function approximation.
For example, the lower-confidence-bound-based algorithms in neural offline contextual bandits (Nguyen-Tang et al., 2022a) and RL (Xu & Liang, 2022) require the inverse of the covariance matrix that is as large as the square of the number of parameters in overparameterized neural networks.
This is computationally prohibited for most practical settings.
Instead, VIPeR avoids such issue while still obtaining provable pessimism and a ˜ O ( 1 √ K ) rate, as shown in the next section.
S UB - OPTIMALITY A NALYSIS
h=1 h= h
=H ... k=
k=K' k=K ...
In this section, we provide a theoretical analysis of the sub-optimality of
VIPeR when the parametric class F in Algorithm 1 is the overparameterized neural network defined in Subsection 3.2.
Our analysis is built on the recent advances in generalization and optimization of deep neural networks (Arora et al., 2019; Allen-Zhu et al., 2019; Hanin & Nica, 2019; Cao & Gu, 2019; Belkin, 2021) that the dynamics of the neural parameters learned by (stochastic) gradient descent can be captured by the corresponding neural tangent kernel (NTK) space (Jacot et al., 2018) when the network is overparameterized.
We define the NTK kernel as follows.
Definition 1 (NTK (Jacot et al., 2018)) .
The NTK kernel K ntk : X ×
X → R is defined as
[Equation]
where σ ′ ( u ) = 1 { u ≥ 0 } .
Let H ntk be the reproducing kernel Hilbert space (RKHS) induced by
K ntk .
Note that K ntk is a universal kernel (Ji et al., 2020), thus H ntk is dense in the space of continuous functions on compact set X = S
× A (Rahimi & Recht, 2008).
In the following, we present the notion of an effective dimension.
Definition 2 (Effective dimension) .
For any h ∈ [ H ] , the effective dimension of the NTK matrix on data { x kh } k ∈I h is defined as
[Equation]
where K h :=
[ K ntk ( x ih , x jh )] i,j ∈I h is the Gram matrix of K ntk on the data { x kh } k ∈I h .
We further define ˜ d := max h ∈
[ H ] ˜ d h .
Remark 1.
Intuitively, the effective dimension ˜ d h measures the number of principle dimensions over which the projection of the data { x kh } k ∈I h in the RKHS H ntk is spread.
It was first introduced by
Valko et al.
(2013) for kernelized contextual bandits
and was subsequently adopted by Yang & Wang (2020) and Zhou et al.
(2020) for kernelized RL and neural contextual bandits, respectively.
The effective dimension is data-dependent and can be bounded by ˜ d ≲ K ′ ( d +1) / (2 d ) in the worst case (see Section B).
We define the function class: Q ∗ :=
{ f ( x ) =
(cid:82) R d c ( w ) T xσ ′ ( w T x ) dw : sup w ∥ c ( w ) ∥ 2 p 0 ( w ) < B } , where c : R d → R d is any function, p 0 is the probability density function of N (0 , I d /d ) , and B is some constant.
We impose the following assumption about the regularity of the underlying MDP under function approximation.
Assumption 5.1 (Completeness) .
For any V : S → [0 , H + 1] and any h ∈ [ H ] , B h V ∈ Q ∗ .
Assumption 5.1 ensures that the Bellman operator B h can be captured by an infinite-width neural network.
This assumption is mild as Q ∗ is a dense subset of H ntk (Gao et al., 2019, Lemma C.1) when B = ∞ , thus Q ∗ is an expressive function class when B is sufficiently large.
Moreover, similar assumptions have been used in many prior provably efficient RL works with function approximation (Cai et al., 2019;
Wang et al., 2020b; Yang et al., 2020; Nguyen-Tang et al., 2022b).
We can now present the suboptimality of the policy ˜ π returned by Algorithm 1.
Recall that we use the initialization scheme defined in Subsection 3.2 throughout the paper.
Fix any δ ∈ (0 , 1) .
Theorem 1.
Set σ h = σ := 1 + λ 1 / 2 B
+ ( H + 1) (cid:20)(cid:113) ˜
d log(1 + K ′ /λ )
+ 2 + 2 log(3 H/δ ) (cid:21) , set m
= poly ( K ′ , H, d, B, ˜ d, λ, δ ) is some high-order polynomial of the problem parameters, λ =
[Equation]
where Φ( · ) is the cumulative distribution function of the standard normal distribution.
Then under Assumption 5.1, with probability at least 1 − MHm − 2 − 2 δ , for any s 1 ∈ S , we have
[Equation]
where Λ h :
=
λI md
+ (cid:80) k
∈I h g
( s kh , a kh ; W 0 ) g ( s kh , a kh ; W 0 )
T ∈ R md × md .
Remark 2.
Theorem 1 shows that the randomized design in our proposed algorithm yields a provable uncertainty quantifier even we do not explicitly maintain any confidence regions in the algorithm.
The implicit pessimism via perturbed rewards induces an extra factor of 1 + (cid:112) 2 log( MSAH/δ ) into the confidence parameter β .
We build upon Theorem 1 to obtain an explicit bound using the following data coverage assumption.
Assumption 5.2 (Optimal-Policy Concentrability)
[Equation]
Assumption 5.2 requires any positive-probability trajectory induced by the optimal policy to be covered by the behavior policy.
This data coverage assumption is significantly milder than the uniform coverage assumptions in many FQI-based offline RL algorithms (Munos & Szepesv´ari, 2008; Chen & Jiang, 2019; Nguyen-Tang et al., 2022b) and is common in pessimism-based algorithms (Rashidinejad et al., 2021; Nguyen-Tang et al., 2022a; Chen & Jiang, 2022; Zhan et al., 2022). [Table]
Theorem 2.
For the parameter setting and the same assumption as in Theorem 1, with probability at least 1 − MHm − 2 − 5 δ ,
[Equation]
where ˜ σ : = σ (1 + (cid:112) 2 log( SAH/δ )) .
Remark 3.
Theorem 2 shows that VIPeR achieves an ˜ O (cid:18) κH
[Equation]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
sub
optimality.
Compared to Yang et al.
(2020), we improve by a factor of K 2 dγ − 1 for some γ ∈ (0 , 1) at the expense of √ H .
When realized to a linear MDP in R d lin , ˜ d
= d lin and our bound reduces into ˜ O (cid:16) κH 5 / 2 d lin √ K (cid:17) which improves the bound ˜
O ( d 3 / 2 lin H 2 / √ K ) of PEVI (Jin et al., 2021,
Corollary 4.6) by a factor of √ d lin .
We provide the result summary and comparison in Table 1 and give a more detailed discussion in Subsection B.1.
E XPERIMENTS
In this section, we empirically evaluate the proposed algorithm VIPeR against several state-of-theart baselines, including PEVI (Jin et al., 2021) which explicitly constructs lower confidence bound (LCB) for pessimism in a linear model (thus we re-name this algorithm as LinLCB for comparison convenience in our experiments), NeuraLCB (Nguyen-Tang et al., 2022a) which explicitly construct
LCB using neural network gradients.
We also include NeuraLCB (Diag), which is NeuraLCB with diagonal approximation for estimating the confidence set as suggested in NeuraLCB (Nguyen-Tang et al., 2022a), LinPER which is VIPeR realized to the linear function approximation instead of neural network function approximation, and NeuralGreedy (LinGreedy, respectively) which uses neural networks (linear models, respectively) to fit into the offline data and act greedily with respect to the estimated state-action value functions without any pessimism.
Note that when the parametric class F in Algorithm 1 is realized to neural networks, we name VIPeR as NeuralPER.
We do not use data split in the experiments.
We provide detailed algorithms of the baselines in Section H.
We evaluated all algorithms in two problem settings: (1) the underlying MDP is a linear MDP whose reward functions and transition kernels are linear in some known feature map (Jin et al., 2020), and (2) the underlying MDP is non-linear with horizon length H = 1 (i.e. non-linear contextual bandits)
(Zhou et al., 2020), where the reward functions either synthetic or constructed from MNIST dataset (LeCun et al., 1998).
We also evaluated our algorithm variant and showed its strong performance advantage in the D4RL benchmark (Fu et al., 2020) in Section A.3.
We implemented all algorithms in Pytorch (Paszke et al., 2019) on a server with Intel(R)
Xeon(R)
Gold 6248 CPU @
2.50GHz, 755G RAM, and one NVIDIA Tesla V
Volta GPU Accelerator 32GB Graphics Card.
6. L INEAR MDP S
The main purpose of this experiment is to evaluate our design of perturbed rewards for implicit pessimism in Algorithm 1 for offline RL.
We constructed a hard instance of linear MDPs (Yin et al., 2022a; Min et al., 2021).
Due to page limit, we describe in detail the linear MDP in Subsection A.1.
We run over H ∈ { 20 , 30 , 50 , 80 } and report the averaged sub-optimality of LinLCB, LinPER and
LinGreedy in Figure 2.
In Figure 2, we can observe that LinGreedy, which is uncertainty-agnostic, clearly failed to offlinelearn the linear MDP and suffered much higher sub-optimality than pessimism-based algorithms
LinLCB and LinPER.
Moreover, LinLCB outperforms LinPER when K is smaller than aroundsamples but they match eventually for larger sample size.
Note that unlike LinLCB, LinPER does not construct any confidence regions nor involve inverse of a large covariance matrix and that the y-axis is in log scale thus LinPER already has small sub-optimality in the first K ≈ 400 samples.
These show the effectiveness of the randomized design for implicit pessimism in Algorithm 1.
[Figure]
The main purpose of this experiment is to evaluate the performance and computational efficiency of the proposed algorithm when neural networks are employed.
For that purpose and for simplicity, we evaluated on contextual bandits which are simply MDPs with one-step horizon, i.e.
H = 1 .
In particular, following Zhou et al.
Nguyen-Tang et al.
(2022a), we use the bandit problems specified by the following reward functions: (a) r ( s, a ) = cos(3 s T θ a ) , (b) r ( s, a ) =
exp( − 10( s T θ a ) 2 ) , where s and θ a are generated uniformly at random from the unit sphere S d − 1 with d = 16 and A = 10 and (c) MNIST, where r ( s, a ) =
if a is the true label of input image s and r ( s, a ) = 0 otherwise.
To predict the value of different actions from the same state s using neural networks, we transform a state s ∈ R d into dA -dimensional vectors s (1)
=
( s, 0 , . . .
, 0) , s (2) =
(0 , s, 0 , . . . , 0) , . . .
, s ( A ) =
, 0 , s ) and train the network to map s ( a ) to r ( s, a ) given a pair of data ( s, a ) .
For NeuralPER, NeuralGreedy, NeuraLCB and NeuraLCB (Diag), we use the same neural network architecture with two hidden layers whose width m = 64 , train the network with Adam optimizer (Kingma & Ba, 2014).
Due to page limit, we present in detail the experimental and hyperparameter setting in Subsection A.2.
The averaged sub-optimality over 5 run times are reported in Figure 3.
In Figure 3, we can observe that algorithms that use a linear model i.e. LinLCB and LinPER significantly underperform neural-based algorithms i.e. NeuralGreedy, NeuraLCB, NeuraLCB (Diag) and NeuralPER, suggesting the crucial impact of neural representation for non-linear problems.
It is also interesting to observe from the experimental results that NeuraLCB does not always outperform its diagonal approximation NeuraLCB (Diag) (e.g. in Figure 3.b), triggering a new question on the empirical effectiveness of NTK-based uncertainty for offline RL.
Finally, NeuralPER outperforms all algorithms in the tested benchmarks, suggesting the effectiveness of our randomized design with neural function approximation.
[Figure]
Figure 4 shows the average running time for action selection of neural-based algorithms NeuraLCB, NeuraLCB (Diag), and NeuralPER.
We can observe that algorithms that use explicit confidence regions i.e. NeuraLCB and NeuraLCB (Diag) take significantly much time on selecting an action when either the number of offline samples K or the network width m increase.
This can be foreseeable as to sample an action, NeuraLCB and NeuraLCB (Diag) need to compute the inverse of a large covariance matrix based on the offline data and maintain the confidence region for each action per state.
The diagonal approximation significantly reduces the running time of NeuraLCB but still has the running time scaled with the number of samples and the network width while the running time for action selection of NeuralPER is constant.
As NeuraLCB, NeuraLCB (Diag), and NeuralPER use the same neural network architecture, the running time spent in training one model is similar.
The only difference is that NeuralPER trains M models while NeuraLCB and NeuraLCB (Diag) train one single model.
However, as the perturbed data in Algorithm 1 are independent, training M models in NeuralPER is embarrassingly parallelizable.
In Figure 5, we show the performance of NeuralPER when M changes in { 1 , 2 , 5 , 10 , 20 , 30 , 50 , 100 , 200 } at fixed K = 1000 .
We can see that the sub-optimality of NeuralPER graciously decreases with larger values of M .
The grid search from the previous experiment for Figure 3 also yields M = 10 and M = 20 from the search space M ∈ { 1 , 10 , 20 } as the best result.
This consistently suggests the crucial impact of our algorithmic design using the minimum of the ensemble to obtain pessimism for offline RL.
C ONCLUSION
We propose a novel algorithm that combines randomized value functions with pessimism principle for offline RL.
As a result, our algorithm eliminates the computational overhead of explicitly maintaining a valid confidence region and the inverse of a large covariance matrix for pessimism.
We show that our algorithm yields a provable pessimism and obtains an ˜ O (cid:16) κH 5 / 2 ˜ d √ K (cid:17) sub-optimality.
Extensive experiments demonstrate the strong advantage about computational efficiency and effectiveness of our algorithm for offline RL.
A E XPERIMENT D ETAILS A. L INEAR MDP S
In this subsection, we provide further details to the experiment setup used in Subsection 6.1.
We describe in detail a variant of the hard instance of linear MDPs (Yin et al., 2022a) used in our experiment.
The linear MDP has S
= { 0 , 1 } ,
A = { 0 , 1 , · · · , 99 } , and the feature dimension d = 10 .
Each action a ∈ [99] = { 1 , . .
. , 99 } is represented by its binary encoding vector u a ∈ Rwith entry being either − 1 or 1 .
The feature mapping ϕ
( s, a ) is given by ϕ ( s, a ) =
[ u Ta , δ ( s, a ) , 1 − δ ( s, a )] T ∈ R 10 , where δ ( s, a ) = 1 if ( s, a ) =
(0 , 0) and δ ( s, a ) =
otherwise.
The true measure ν h ( s ) is given by ν h ( s ) =
[0 , · · · , 0 , (1 − s )
⊕ α h , s ⊕
α h ] where { α h } h ∈ [ H ] ∈ { 0 , 1 } H are generated uniformly at random and ⊕ is the XOR operator.
We define θ h = [0 , · · · , 0 , r, 1 − r ] T ∈ R 10 where r = 0 . 99 .
Recall that the transition follows P h ( s ′ | s, a ) = ⟨ ϕ ( s, a ) , ν h ( s ′ ) ⟩ and the mean reward r h ( s, a ) = ⟨ ϕ ( s, a ) , θ h ⟩ .
We generated a priori K ∈ { 1 , . . .
, 1000 } trajectories using the behavior policy µ , where for any h ∈
[ H ] we set µ h (0 | 0) =
p, µ h (1 | 0)
= 1 − p, µ h ( a | 0) = 0 ,
∀ a > 1; µ h (0 | 1) = p, µ h ( a | 1) =
(1 − p ) / 99 , ∀ a > 0 , where we set p = 0 . 6 .
We run over K ∈ { 1 , . .
. , 1000 } and H ∈ { 20 , 30 , 50 , 80 } .
We set λ = 0 .
01 for all algorithms.
For
LinPER, we grid searched σ h
= σ ∈ { 0 . 0 , 0 . 1 , 0 . 5 , 1 . 0 , 2 . 0 } and M ∈ { 1 , 2 , 10 , 20 } .
For LinLCB, we grid searched its uncertainty multiplier β ∈ { 0 . 1 , 0 .
The sub-optimality metric is used to compare algorithms.
For each H ∈ { 20 , 30 , 50 , 80 } , each algorithm was executed for 30 times and the averaged results (with std) are reported in Figure 2.
A. N EURAL C ONTEXTUAL B ANDITS
In this subsection, we provide in detail the experimental and hyperparameter setup in our experiment in Subsection 6.2.
For NeuralPER, NeuralGreedy, NeuraLCB and NeuraLCB (Diag), we use the same neural network architecture with two hidden layers whose width m = 64 , train the network with Adam optimizer (Kingma & Ba, 2014) with learning rate being grid-searched over { 0 . 0001 , 0 . 001 , 0 . 01 } and batch size of 64 .
For NeuraLCB, NeuraLCB (Diag), and LinLCB, we grid-searched β over { 0 . 001 , 0 . 01 , 0 . 1 , 1 , 5 , 10 } .
For NeuralPER and LinPER, we grid-searched σ h
= σ over { 0 . 001 , 0 . 01 , 0 . 1 , 1 , 5 , 10 } and M over { 1 , 10 , 20 } .
We did not run NeuraLCB in
MNIST as the inverse of a full covariance matrix in this case is extremely expensive.
We fixed the regularization parameter λ = 0 .
01 for all algorithms.
Offline data is generated by the (1 − ϵ ) -optimal policy which generates non-optimal actions with probability ϵ and optimal actions with probability 1 − ϵ .
We set ϵ = 0 .
5 in our experiments.
To estimate the expected sub-optimality, we randomly obtain 1 , 000 novel samples (i.e. not used in training) to compute the average sub-optimality and keep these same samples for all algorithms.
A. E XPERIMENT IN D4RL B ENCHMARK
In this subsection, we evaluate the effectiveness of the reward perturbing design of VIPeR in the
Gym domain in the D4RL benchmark (Fu et al., 2020).
The Gym domain has three environments (HalfCheetah, Hopper, and Walker2d) with five datasets (random, medium, medium-replay, medium-expert, and expert), making up 15 different settings.
Design.
To adapt the design of VIPeR to continuous control, we use the actor-critic framework.
Specifically, we have M critics { Q θ i } i ∈
[ M ] and one actor π ϕ , where { θ i } i ∈
[ M ] and ϕ are the learnable parameters for the critics and actor, respectively.
Note that in the continuous domain, we consider discounted MDP with discount factor γ , instead of finite-time episode MDP as we initially considered in our setting in the main paper.
In the presence of the actor π ϕ , there are two modifications to Algorithm 1.
The first modification is that when training the critics { Q i θ } i ∈ [ M ] , we augment the training loss in Algorithm 2 with a new penalization term.
Specifically, the critic loss for Q θ i on a training sample τ :=
( s, a, r, s ′ )
(sampled from the offline data D ) is
[Equation]
where ¯ θ
i has the same value of the current θ i but is kept fixed, ¯ Q = 1 M (cid:80)
Mi =1 Q θ i and ξ ∼ N (0 , σ 2 ) is Gaussian noise, and β is a penalization parameter (note that β here is totally different from the β in Theorem 1).
The penalization term R ( θ i ; s, ϕ ) discourages overestimation in the value function estimate Q θ i for out-of-distribution (OOD) actions a ′ ∼ π ϕ ( ·| s ) .
Our design of R ( θ i ; s, ϕ ) is initially inspired by the OOD penalization in Bai et al.
(2022) that creates a pessimistic pseudo target for the values at OOD actions.
Note that we do not need any penalization for OOD actions in our experiment for contextual bandits in Section 6.2.
This is because in the contextual bandit setting in Section 6.2 the action space is finite and not large, thus the offline data often sufficiently cover all good actions.
In the continuous domain such as the Gym domain of D4RL, however, it is almost certain that there are actions that are not covered by the offline data since the action space is continuous.
We also note that the inclusion of the OOD action penalization term R ( θ i ; s, ϕ ) in this experiment does not contradict our guarantee in Theorem 1 since in the theorem we consider finite action space while in this experiment we consider continuous action space.
We argue that the inclusion of some regularization for OOD actions (e.g., R ( θ i ; s, ϕ ) ) is necessary for the continuous domain.
The second modification to Algorithm 1 for the continuous domain is the actor training, which is the implementation of policy extraction in line 10 of Algorithm 1.
Specifically, to train the actor π ϕ given the ensemble of critics { Q iθ } i ∈ [ M ] , we use soft actor update in Haarnoja et al.
(2018) via
[Equation]
which is trained using gradient ascent in practice.
Note that in the discrete action domain, we do not need such actor training as we can efficiently extract the greedy policy with respect to the estimated action-value functions when the action space is finite.
Also note that we do not use data splitting and value truncation as in the original design of Algorithm 1.
Hyperparameters.
For the hyper-parameters of our training, we set M = 10 and the noise variance σ = 0 .
For β , we decrease it from 0 .
5 to 0 .
2 by linear decay for the first 50K steps and exponential decay for the remaining steps.
For the other hyperparameters of actor-critic training, we fix them the same as in Bai et al.
Specifically, the Q-network is the fully connected neural network with three hidden layers all of which has 256 neurons.
The learning rate for the actor and the critic are 10 − 4 and 3 × 10 − 4 , respectively.
The optimizer is Adam.
Results.
We compare VIPeR with several state-of-the-art algorithms, including (i) BEAR (Kumar et al., 2019) that use MMD distance to constraint policy to the offline data, (ii) UWAC (Wu et al., 2021) that improves BEAR using dropout uncertainty, (iii) CQL (Kumar et al., 2020) that minimizes Q-values of OOD actions, (iv) MOPO (Yu et al., 2020) that uses model-based uncertainty via ensemble dynamics, (v) TD3-BC (Fujimoto & Gu, 2021) that uses adaptive behavior cloning, and (vi) PBRL (Bai et al., 2022) that use uncertainty quantification via disagreement of bootstrapped
Q-functions.
We follow the evaluation protocol in Bai et al.
We run our algorithm for five seeds and report the average final evaluation scores with standard deviation.
We report the scores of our method and the baselines in Table 2.
We can see that our method has a strong advantage of good performance (highest scores) in 11 out of 15 settings, and has good stability (small std) in all settings.
Overall, we also have the strongest average scores aggregated over all settings.
B E XTENDED D ISCUSSION
Here we provide extended discussion of our result.
B. C OMPARISON WITH OTHER WORKS AND DISCUSSION
We provide further discussion regarding comparison with other works in the literature.
[Table]
Comparing to Jin et al.
When the underlying MDP reduces into a linear MDP, if we use the linear model as the plug-in parametric model in Algorithm 1, our bound reduces into ˜ O (cid:16) κH 5 / 2 d lin √ K (cid:17) which improves the bound ˜
O ( d 3 / 2 lin H 2 / √ K ) of PEVI (Jin et al., 2021,
Corollary 4.6) by a factor of √ d lin and worsen by a factor of √ H due to the data splitting.
Thus, our bound is more favorable in the linear MDPs with high-dimensional features.
Moreover, our bound is guaranteed in more practical scenarios where the offline data can have been adaptively generated and is not required to uniformly cover the state-action space.
The explicit bound ˜ O
( d 3 / 2 lin H 2 / √ K ) of PEVI (Jin et al., 2021, Corollary 4.6) is obtained under the assumption that the offline data have uniform coverage and are generated independently on the episode basis.
Comparing to Yang et al.
Though Yang et al.
(2020) work in the online regime, it shares some part of the literature with our work in function approximation for RL.
Besides different learning regimes (offline versus online), we offer three key distinctions which can potentially be used in the online regime as well: (i) perturbed rewards, (ii) optimization, and (iii) data split.
Regarding (i), our perturbed reward design can be applied to online RL with function approximation to obtain a provably efficient online RL that is computationally efficient and thus remove the need of maintaining explicit confidence regions and performing the inverse of a large covariance matrix.
Regarding (ii), we incorporate the optimization analysis into our algorithm which makes our algorithm and analysis more practical.
We also note that unlike (Yang et al., 2020), we do not make any assumption on the eigenvalue decay rate of the empirical NTK kernel as the empirical NTK kernel is data-dependent.
Regarding (iii), our data split technique completely removes the factor (cid:112) log N ∞ ( H , 1 /K, B ) in the bound at the expense of increasing the bound by a factor of √ H .
In complex models, such log covering number can be excessively larger than the horizon H , making the algorithm too optimistic in the online regime (optimistic in the offline regime, respectively).
For example, the target function class is RKHS with a γ -polynomial decay, the log covering number scales as (Yang et al., 2020, Lemma D1),
[Equation]
for some α ∈ (0 , 1) .
In the case of two-layer ReLU NTK, γ = d (Bietti & Mairal, 2019), thus (cid:112) log N [Equation] [Equation] 1 which is much larger than √ H when the size of dataset is large.
Note that our data-splitting technique is general that can be used in the online regime as well.
Comparing to Xu & Liang (2022).
Xu & Liang (2022) consider a different setting where pertimestep rewards are not available and only the total reward of the whole trajectory is given.
Used with neural function approximation, they obtain ˜ O
( D eff H 2 / √ K ) where D eff is their effective dimension.
Note that Xu & Liang (2022) do not use data splitting and still achieve the same order of
D eff as our result with data splitting.
It at first might appear that our bound is inferior to their bound as we pay the cost of √ H due to data splitting.
However, to obtain that bound, they make three critical assumptions: (i) the offline data trajectories are independently and identically distributed (i.i.d.)
(see their Assumption 3), (ii) the offline data is uniformly explorative over all dimensions of the feature space (also see their Assumption 3), and (iii) the eigenfunctions of the induced NTK RKHS has finite spectrum (see their Assumption 4).
The i.i.d. assumption under the RKHS space with finite dimensions (due to the finite spectrum assumption) and the well-explored dataset is critical in their proof to use a matrix concentration that does not incur an extra factor of √ D eff as it would normally do without these assumptions (see Section E, the proof of their Lemma 2).
Note that the celebrated
ReLU NTK does not satisfy the finite spectrum assumption (Bietti & Mairal, 2019).
Moreover, we do not make any of these three assumptions above for our bound to hold.
That suggests that our bound is much more general.
In addition, we do not need to compute any confidence regions nor perform the inverse of a large covariance matrix.
Comparing to Yin et al.
(2022b).
During the submission of our work, a concurrent work of Yin et al.
(2022b) appeared online.
Yin et al.
(2022b) study provably efficient offline RL with a general parametric function approximation that unifies the guarantees of offline RL in linear and generalized linear MDPs, and beyond with potential applications to other classes of functions in practice.
We remark that the result in Yin et al.
(2022b) is orthogonal/complementary to our paper since they consider the parametric class with third-time differentiability which cannot apply to neural networks (not necessarily overparameterized) with non-smooth activation such as ReLU.
In addition, they do not consider reward perturbing in their algorithmic design or optimization errors in their analysis.
B. W ORSE -C ASE R ATE OF E FFECTIVE D IMENSION
In the main paper, we prove an ˜ O (cid:16) κH 5 / 2 ˜ d √ K (cid:17) sub-optimality bound which depends on the notion of effective dimension defined in Definition 2.
Here we give a worst-case rate of the effective dimension ˜ d for the two-layer ReLU NTK.
We first briefly review the background of RKHS.
Let H be an RKHS defined on X ⊆ R d with kernel function ρ : X ×X → R .
Let ⟨· , ·⟩ H : H×H
→ R and ∥ · ∥ H : H → R be the inner product and the RKSH norm on H .
By the reproducing kernel property of H , there exists a feature mapping ϕ :
X → H such that f ( x ) = ⟨ f, ϕ ( x ) ⟩ H and ρ ( x, x ′ ) =
⟨ ϕ ( x ) , ϕ ( x ′ ) ⟩ H .
We assume that the kernel function ρ is uniformly bounded, i.e. sup x ∈X ρ ( x, x ) < ∞ .
Let L 2 ( X ) be the space of square-integral functions on X with respect to the
Lebesgue measure and let ⟨· , ·⟩ L 2 be the inner product on L 2 ( X ) .
The kernel function ρ induces an integral operator T ρ : L 2 ( X ) →
L 2 ( X ) defined as
[Equation]
By Mercer’s theorem (Steinwart & Christmann, 2008), T ρ has countable and positive eigenvalues { λ i } i ≥ 1 and eigenfunctions { ν i } i ≥ 1 .
The kernel function and H can be expressed as
[Equation]
Now consider the NTK defined in Definition 1:
[Equation]
It follows from (Bietti & Mairal, 2019, Proposition 1) that λ
i ≍ i − d .
Thus, by (Srinivas et al., 2009,
Theorem 5), the data-dependent effective dimension of H ntk can be bounded in the worst case by
[Equation]
d ≲ K
[Equation]
d +
[Equation]
We remark that this is the worst-case bound that considers uniformly over all possible realizable of training data.
The effective dimension ˜ d is on the other hand data-dependent, i.e. its value depends on the specific training data at hand thus ˜ d can be actually much smaller than the worst-case rate.
C P ROOF OF T HEOREM 1 AND T HEOREM
In this section, we provide both the outline and detailed proofs of Theorem 1 and Theorem 2.
C. T ECHNICAL R EVIEW AND P ROOF O VERVIEW
Technical Review.
In what follows, we provide more detailed discussion when placing our technical contribution in the context of the related literature.
Our technical result starts with the value difference lemma in Jin et al.
(2021) to connect bounding the suboptimality of an offline algorithm to controlling the uncertainty quantification in the value estimates.
Thus, our key technical contribution is to provably quantify the uncertainty of the perturbed value function estimates which were obtained via reward perturbing and gradient descent.
This problem setting is largely different from the current analysis of overparameterized neural networks for supervised learning which does not require uncertainty quantification.
Our work is not the first to consider uncertainty quantification with overparameterized neural networks, since it has been studied in Zhou et al.
Nguyen-Tang et al.
(2022a)
; Jia et al.
However, there are significant technical differences between our work and these works.
The work in
Zhou et al.
Nguyen-Tang et al.
(2022a) considers contextual bandits with overparameterized neural networks trained by (S)GD and quantifies the uncertainty of the value function with explicit empirical covariance matrices.
We consider general MDP and use reward perturbing to implicitly obtain uncertainty, thus requiring different proof techniques.
Jia et al.
(2022) is more related to our work since they consider reward perturbing with overparameterized neural networks (but they consider contextual bandits).
However, our reward perturbing strategy is largely different from that in Jia et al.
Specifically, Jia et al.
(2022) perturbs each reward only once while we perturb each reward multiple times, where the number of perturbing times is crucial in our work and needs to be controlled carefully.
We show in Theorem 1 that our reward perturbing strategy is effective in enforcing sufficient pessimism for offline learning in general
MDP and the empirical results in Figure 2, Figure 3, Figure 5, and Table 2 are strongly consistent with our theoretical suggestion.
Thus, our technical proofs are largely different from those of Jia et al.
Finally, the idea of perturbing rewards multiple times in our algorithm is inspired by Ishfaq et al.
However, Ishfaq et al.
(2021) consider reward perturbing for obtaining optimism in online
RL.
While perturbing rewards are intuitive to obtain optimism for online RL, for offline RL, under distributional shift, it can be paradoxically difficult to properly obtain pessimism with randomization and ensemble (Ghasemipour et al., 2022), especially with neural function approximation.
We show affirmatively in our work that simply taking the minimum of the randomized value functions after perturbing rewards multiple times is sufficient to obtain provable pessimism for offline RL.
In addition, Ishfaq et al.
(2021) do not consider neural network function approximation and optimization.
Controlling the uncertainty of randomization (via reward perturbing) under neural networks with extra optimization errors induced by gradient descent sets our technical proof significantly apart from that of Ishfaq et al.
Besides all these differences, in this work, we propose an intricately-designed data splitting technique that avoids the uniform convergence argument and could be of independent interest for studying sample-efficient RL with complex function approximation.
Proof Overview.
The key steps for proving Theorem 1 and Theorem 2 are highlighted in Subsection C.2 and Subsection C.3, respectively.
Here, we discuss an overview of our proof strategy.
The key technical challenge in our proof is to quantify the uncertainty of the perturbed value function estimates.
To deal with this, we carefully control both the near-linearity of neural networks in the
NTK regime and the estimation error induced by reward perturbing.
A key result that we use to control the linear approximation to the value function estimates is Lemma D.3.
The technical challenge [Table] in establishing Lemma D.3 is how to carefully control and propagate the optimization error incurred by gradient descent.
The complete proof of Lemma D.3 is provided in Section E.3.
The implicit uncertainty quantifier induced by the reward perturbing is established in Lemma D. and Lemma D.2, where we carefully design a series of intricate auxiliary loss functions and establish the anti-concentrability of the perturbed value function estimates.
This requires a careful design of the variance of the noises injected into the rewards.
To deal with removing a potentially large covering number when we quantify the implicit uncertainty, we propose our data splitting technique which is validated in the proof of Lemma D.1 in
Section E.1.
Moreover, establishing Lemma D.1 in the overparameterization regime induces an additional challenge since a standard analysis would result in a vacuous bound that scales with the overparameterization.
We avoid this issue by carefully incorporating the use of the effective dimension in Lemma D.1.
C. P ROOF OF T HEOREM
In this subsection, we present the proof of Theorem 1.
We first decompose the suboptimality
SubOpt(˜ π ; s ) and present the main lemmas to bound the evaluation error and the summation of the implicit confidence terms, respectively.
The detailed proof of these lemmas are deferred to Section
D. For proof convenience, we first provide the key parameters that we use consistently throughout our proofs in Table 3.
We define the model evaluation error at any ( x, h ) ∈
X ×
[ H ] as
[Equation]
where B h is the Bellman operator defined in Section 3, and ˜ V h and ˜ Q
h are the estimated (action-) state value functions returned by Algorithm 1.
Using the standard suboptimality decomposition (Jin et al., 2021, Lemma 3.1), for any s 1 ∈ S ,
[Equation]
where the third term is non-positive as ˜ π h is greedy with respect to ˜ Q h .
Thus, for any s 1 ∈ S , we have
[Equation]
In the following main lemma, we bound the evaluation error err h ( s, a ) .
In the rest of the proof, we consider an additional parameter R and fix any δ ∈ (0 , 1) .
Lemma C.1.
Let   [Equation] [Equation] λ −
[Equation]
H + ψ + γ h
[Equation]
[Equation]
+
[Equation]
γ
[Equation]
h
[Equation]
[Equation]
η ≤ (
[Equation]
+ K ′ C
[Equation]
g ) −
[Equation]
ψ > ι, σ h ≥ β
[Equation]
∀ h ∈
[ H ] [Equation] [Equation] where ˜ B 1 , ˜ B 2 , γ h, 1 , γ h, 2 , and ι are defined in Table 3, C g is a absolute constant given in Lemma G.1, and R is an additional parameter.
Let M = log HSAδ / log 1 1 − Φ( − 1) where Φ( · ) is the cumulative distribution function of the standard normal distribution.
With probability at least 1 − MHm − 2 − 2 δ , for any ( x, h ) ∈ X ×
[ H ] , we have
[Equation]
where Λ
[Equation]
[Equation]
Now we can prove Theorem 1.
Proof of Theorem 1.
Theorem 1 can directly follow from substituting Lemma C.1 into Equation (4).
We now only need to simplify the conditions in Equation (5).
To satisfy Equation (5), it suffices to set 
[Equation]
Combining with Equation 5, we have  
[Equation]
Note that with the above choice of λ =
+ HK , we have
[Equation]
We further set that m ≳
B 2 K ′
d log(3 H/δ ) , we have
[Equation]
Thus, 4 √ K ′
[Equation]
for K ′ large enough.
Therefore
[Equation]
there exists R that satisfies Equation (6).
We now only need to verify ι < 1 .
We have
[Equation]
[Equation]
if
[Equation]
Note that
[Equation]
Thus [Equation] Equation (7) is satisfied if
[Equation]
Finally note that ι 2 ≤ ι 1 .
Rearranging the derived conditions here gives the complete parameter conditions in Theorem 1.
Specifically
[Equation]
the polynomial form of m is m ≳ max { R 8 log 3 m, K ′ 10 ( H +
[Equation]
C. P ROOF OF T HEOREM
In this subsection [Equation] we give a detailed proof of Theorem 2.
We first present intermediate lemmas whose proofs are deferred to Section D. For any h ∈
[ H ] and k ∈ I h
=
[( H − h ) K ′ + 1 , . .
, ( H − h + 1) K ′ ] , we define the filtration
[Equation]
Let
[Equation]
In the following lemma [Equation] we connect the expected sub-optimality of ˜ π to the summation of the uncertainty quantifier at empirical data.
Lemma C.2.
Suppose that the conditions in Theorem 1 all hold.
With probability at least 1 −
MHm − 2 − 3 δ ,
SubOpt
[Equation]
Lemma C.3.
Under Assumption 5.2, for any h ∈ [ H ] and fixed W 0 , with probability at least 1 − δ , (cid:88) k ∈I
[Equation]
Lemma C.4.
If λ ≥ C 2 g and m = Ω( K ′ 4 log( K ′ H/δ )) , then with probability at least 1 − δ , for any h ∈
[ H ] , we have
[Equation]
where ˜ d h is the effective dimension defined in Definition 2.
Proof of Theorem 2.
Theorem 2 directly follows from Lemma C.2-C.3-C.4 using the union bound.
D P ROOF OF L EMMA C.
In this section, we provide the proof for Lemma C.1.
We set up preparation for all the results in the rest of the paper and provide intermediate lemmas that we use to prove Lemma C.1.
The detailed proofs of these intermediate lemmas are deferred to Section E.
D. P REPARATION
To prepare for the lemmas and proofs in the rest of the paper, we define the following quantities.
Recall that we use abbreviation x = ( s, a ) ∈
X ⊂ S d − 1 and x kh =
( s kh , a kh ) ∈
X ⊂ S d − 1 .
For any h ∈ [ H ] and i ∈
[ M ] , we define the perturbed loss function
[Equation]
where
[Equation]
˜ V h +1 is computed by Algorithm 1 at Line 10 for timestep h + 1 , and { ξ i,kh } and ζ ih are the Gaussian noises obtained at Line 5 of Algorithm 1.
Here the subscript h and the superscript i in ˜ L ih ( W ) emphasize the dependence on the ensemble sample i and timestep h .
The gradient descent update rule of ˜ L ih ( W ) is
[Equation]
[Equation]
where ˜ W i, (0) h
= W 0 is the initialization parameters.
Note that
[Equation]
where ˜ W ih is returned by Line 7 of Algorithm 1.
We consider a non-perturbed auxiliary loss function
[Equation]
where
[Equation]
Note that L h ( W ) is simply a non-perturbed version of ˜ L ih ( W ) where we drop all the noises { ξ i,kh } and { ζ ih } .
We consider the gradient update rule for L h ( W ) as follows
[Equation]
where ˆ W (0) h
= W 0 is the initialization parameters.
To correspond with ˜ W ih , we denote
[Equation] [Equation]
We also define the auxiliary loss functions for both non-perturbed and perturbed data in the linear model with feature g ( · ; W 0 ) as follows
[Equation]
We consider the auxiliary gradient updates for ˜ L i,linh ( W ) as
[Equation]
[Equation] [Equation]
where ˜ W i,lin, (0)
h
= ˆ W i,lin, (0) h
= W 0 for all i, h .
Finally, we define the least-square solutions to the auxiliary perturbed and non-perturbed loss functions for the linear model as follows
[Equation]
For any h ∈ [ H ] , we define the auxiliary covariance matrix Λ h as follows
[Equation]
It is worth remarking that Algorithm 1 only uses Equation (8) and (9) thus it does not actually require any of the auxiliary quantities defined in this subsection during its run time.
The auxiliary quantities here are only for our theoretical analysis.
D. P ROOF OF L EMMA C.
In this subsection, we give detailed proof of Lemma C.1.
To prepare for proving Lemma C.1, we first provide the following intermediate lemmas.
The detailed proofs of these intermediate lemmas are deferred to Section E.
In the following lemma, we bound the uncertainty f
( x ; ˆ W h ) in estimating the Bellman operator at the estimated state-value function B h ˜ V h
+1 .
Lemma D.1.
Let
[Equation]
With probability at least 1 − Hm − 2 − 2 δ , for any x ∈ S d − 1 , and any h ∈ [ H ] ,
[Equation]
where ˜ V h +1 is computed by Algorithm 1 for timestep h + 1 , ˆ W h is defined in Equation (12), and β , ι 2 and ι 0 are defined in Table 3.
In the following lemma, we establish the anti-concentration of ˜ Q h .
Lemma D.2.
Let
[Equation]
where ˜ B 1 , ˜ B 2 , γ h, 1 and γ h, 2 are defined in Table 3, and C g is a constant given in Lemma G.1.
Let
M = log HSAδ / log 1 1 − Φ( − 1) where Φ( · ) is the cumulative distribution function of the standard normal distribution and M is the number of bootstrapped samples in Algorithm 1.
Then with probability 1 − MHm − 2 − δ , for any x ∈ S d − 1 and h ∈ [ H ] ,
[Equation]
where ˆ W linh is defined in Equation (18), ˜ Q h is computed by Line 9 of Algorithm 1, and ι 1 and ι 2 are defined in Table 3.
We prove the following linear approximation error lemma.
Lemma D.3.
Let
[Equation]
where ˜ B 1 , ˜ B 2 , γ h, 1 and γ h, 2 are defined in Table 3, and C g is a constant given in Lemma G.1.
With probability at least 1 − MHm − 2 − δ , for any ( x, i, j, h ) ∈ S d − 1 ×
[ M ] × [
[Equation]
where ˜ W i, ( j ) h , ˜ W i,linh , and ι 1 are defined in Equation (9), Equation (17), and Table 3, respectively.
In addition, with probability at least 1 − Hm − 2 , for any for any ( x, j, h ) ∈
S
d − 1 ×
[ J ] × [ H ] ,
[Equation]
where ˆ W ( j ) h , ˆ W linh , and ι 2 are defined in Equation (11), Equation (18), and Table 3, respectively.
We now can prove Lemma C.1.
Proof of Lemma C.1.
Note that the first fourth conditions in Equation (5) of Lemma C.1 satisfy
Equation (21).
Moreover, the event in which the inequality in Lemma D.3 holds already implies the event in which the inequality in Lemma D.1 holds (see the proofs of Lemma D.3 and Lemma D. in Section D).
Now in the rest of the proof, we consider the joint event in which both the inequality of Lemma D.3 and that of Lemma D.1 hold.
Then, we also have the inequality in Lemma D.1.
Consider any x ∈ X ,
h ∈
[ H ] .
It follows from Lemma D.1 that
[Equation]
It follows from Lemma D.2 that
[Equation]
Note that ˜ Q h ( x ) ≥ 0 .
If ⟨ g ( x ; W 0 ) , ˆ W linh − W 0 ⟩ − σ h ∥ g ( x ; W 0 )
∥ Λ − 1 h + ι 1 + ι 2 ≤ 0 , Equation (23) implies that ˜ Q h ( x ) = 0 and thus
[Equation]
Otherwise, if ⟨ g ( x ; W 0 ) , ˆ W linh − W
[Equation]
∥ g ( x ; W 0 ) ∥ Λ − 1 h + ι 1 + ι 2 > 0 , Equation (23) implies that [Equation]
[Equation]
Thus, combining Equation (22), (24) and Lemma D.3, with the choice σ h ≥ β , we have
[Equation]
As ι ≥ 0 , in either case, we have
[Equation]
Note that due to Equation (25), we have
[Equation]
where the last inequality holds due to the choice ψ > ι .
Thus, we have
[Equation] [Equation]
Substituting Equation (26) into the definition of err h ( x ) , we have
[Equation]
where the first inequality holds due to Equation (26), the second inequality holds due to Lemma D.1, the third inequality holds due to Lemma D.3, and the last inequality holds due to Lemma E.2 and
Lemma G.3 via the union bound.
D. P ROOF OF L EMMA C.
Proof of Lemma C.2.
Let Z
[Equation]
W
[Equation]
(cid:105) where 1 {} is the indicator function.
Under the event in which the inequality in Theorem 1 holds, we have
[Equation]
where the first inequality holds due to Theorem 1 and that SubOpt(˜ π ; s 1 ) ≤ H, ∀ s 1 ∈ S , the second inequality holds due to min { a, b + c } ≤ min { a, b } + c , the third inequality holds due to that Λ − 1 h ⪯ (Λ kh ) − 1 , the fourth inequality holds due to Jensen’s inequality for the convex function f ( x ) =
min { H, x } .
It follows from Lemma G.9 that with probability at least 1 − δ ,
[Equation]
Substituting Equation (28) into Equation (27) and using the union bound complete the proof.
D. P ROOF OF L EMMA C.
Proof of Lemma C.3.
Let Z
kh := 1 { k ∈
I
h
} d ∗ h ( x kh )
d µh ( x kh )
∥ g ( x kh ; W 0 ) ∥
(Λ kh ) − 1 .
We have Z kh is F khmeasurable, and by Assumption 5.2, we have,
[Equation]
Thus, by Lemma G.
[Equation]
for any h ∈ [ H ]
[Equation]
with probability at least 1 − δ
[Equation]
we have:
K (cid:88)
[Equation]
≤
K
[Equation]
=
[Equation]
D. P ROOF OF L EMMA C.
Proof of Lemma C.4.
For any fixed h ∈ [ H ] , let
U = [ g ( x kh ; W 0 )] k ∈I h ∈ R md × K ′ .
By the union bound, with probability at least 1 − δ , for any h ∈ [ H ] , we have (cid:88) k ∈I h [Equation] =
[Equation]
=
[Equation]
=
[Equation]
) ≤
[Equation]
h /λ ) + 2 tr (cid:0) ( I + K h /λ ) − 1 ( U T U − K h ) /λ (cid:1) ≤ 2 logdet( I + K h /λ )
+ 2 ∥
( I + K h /λ ) − 1 ∥ F ∥ U T U − K h ∥ F ≤ 2 logdet( I + K h /λ ) + 2 √ K ′ ∥ U T U − K h ∥ F ≤ 2 logdet( I + K h /λ ) += 2 ˜
d h log(1 + K ′ /λ ) + where the first inequality holds due to λ ≥ C 2 g and (Abbasi-yadkori et al., 2011, Lemma 11), the third equality holds due to that logdet( I + AA T ) = logdet( I + A T A ) , the second inequality holds due to that logdet( A + B ) ≤ logdet( A ) + tr( A − 1 B ) as the result of the convexity of logdet , the third inequality holds due to that tr( A ) ≤ ∥
A ∥ F , the fourth inequality holds due to 2 √ K ′ ∥
U T U −
K h ∥ F ≤ 1 by the choice of m = Ω
( K ′ 4 log( K ′ H/δ )) , Lemma G.2 and the union bound, and the last equality holds due to the definition of ˜ d h .
E P ROOFS OF L EMMAS IN S ECTION D E. P ROOF OF L EMMA D.
In this subsection, we give detailed proof of Lemma D.1.
For this, we first provide a lemma about the linear approximation of the Bellman operator.
In the following lemma, we show that B h ˜ V h
+ can be well approximated by the class of linear functions with features g ( · ; W 0 ) with respect to l ∞ -norm.
Lemma E.1.
Under Assumption 5.1, with probability at least 1 − δ over w 1 , . . .
, w m drawn i.i.d.
from N (0 , I d ) , for any h ∈ [ H ] , there exist c 1 , . . .
, c m where c i ∈ R d and ∥ c i ∥ 2 ≤ B m such that
[Equation]
Moreover, ¯ Q h ( x ) can be re-written as
[Equation]
1 c T 1 , . . .
, a m c Tm ] T ∈ R
md , and ∥ ¯ W h ∥ 2 ≤ B.
We now can prove Lemma D.1.
Proof of Lemma D.1.
We first bound the difference ⟨ g ( x ; W 0 ) , ¯ W h ⟩ − ⟨ g ( x ; W 0 ) , ˆ W linh − W 0 ⟩ : ⟨ g ( x ; W 0 ) , ¯ W h ⟩ − ⟨ g ( x ; W 0 ) ,
[Equation]
For bounding I 1 , it follows from Lemma E.1 that with probability at least 1 − δ/ 3 , for any for any x ∈ S
d − 1 and any h ∈ [ H ] , | ( B h ˜ V h +1 )
( x ) − ⟨ g ( x ; W 0 ) , ¯ W h ⟩| ≤ ι 0 , where ι 0 is defined in Table 3.
where ¯ W h is defined in Lemma E.1.
Thus, with probability at least 1 − δ/ 3 , for any for any x ∈ S d − 1 and any h ∈ [ H ] ,
[Equation]
[Table]
[Table]
k
[Table]
λg
[Table]
[Equation]
where the first equation holds due to the definition of Λ h
[Equation]
and the last inequality holds due to Step I with ∥ ¯ W
[Equation]
For bounding I 2 , we have
[Equation]
If we directly apply the result of Jin et al.
(2021) in linear MDP
[Equation]
we would get
[Equation]
which gives a vacuous bound as m is sufficiently larger than K in our problem.
Instead, in the following, we present an alternate proof that avoids such vacuous bound.
For notational simplicity, we write
[Equation]
We denote K inith :
= [ ⟨ g ( x ih ; W 0 ) , g ( x jh ; W 0 ) ⟩ ] i,j ∈I h as the Gram matrix of the empirical NTK kernel on the data { x kh } k ∈ [ K ] .
We denote
[Equation]
Recall the definition of the Gram matrix K h of the NTK kernel on the data { x kh } k ∈I h .
It follows from Lemma G.2 and the union bound that if m = Ω( ϵ − 4 log(3 K ′ H/δ )) with probability at least 1 − δ/ 3 , for any h ∈
[ H ] ,
[Equation]
We now can bound I 3 .
We have
I
[Equation]
[Equation]
We bound each I
and I 5 separately.
For bounding I 4 , applying Lemma G.1, with 1 − Hm − 2 , for any h ∈ [ H ] ,
[Equation]
[Equation] 34) where the first inequality holds due to the triangle inequality, the second inequality holds due to the triangle inequality, Lemma G.7, and ∥ ( K h + λI K ′ )
− 1 ∥ 2 ≤ λ − 1 , the third inequality holds due to ∥K inith ∥ 2 ≤ ∥ G 0 ∥ 22 ≤ ∥ G
∥ 2 F ≤ K ′ C 2 g due to Lemma G.1, the fourth inequality holds due to ∥ E h ∥ 2 ≤ √ K ′ ( H + ψ ) , λ ≥ 1 , and K ′ C 2 g ≥ λ .
Substituting Equation (32) in Equation (34) using the union bound, with probability 1 −
Hm − 2 − δ/ 3 , for any h ∈
[ H ] ,
[Equation]
35) where the last inequality holds due to the choice of ϵ = 1 / 2 K ′− 5 / 2 ( H + ψ ) − 2 C − 2 g and thus
[Equation]
For bounding I 5 , as λ > 1 , we have
[Equation]
Let σ ( · ) be the σ -algebra induced by the set of random variables.
For any h ∈ [ H ] and k ∈ I h
=
[( H − h ) K ′ + 1 , . .
, ( H − h + 1) K ′ ] , we define the filtration
[Equation]
which is simply all the data up to episode k + 1 and timestep h
but right before r k +1 h
and s k +1 h +1 are generated (in the offline data).
7 Note that for any k ∈
I h , we have ( s kh , a kh , r kh , s [-k h +
-]kh +1 ) ∈ F kh , and
[Equation]
Thus, for any k ∈ I h , we have
[Equation]
The key property in our data split design is that we nicely have that
[Equation]
Thus, conditioned on F k − 1 h , ˜ V
h +1 becomes deterministic.
This implies that
[Equation]
Note that this is only possible with our data splitting technique.
Otherwise, ϵ kh is not zero-mean due to the data dependence structure induced in offline RL with function approximation (Nguyen-Tang et al., 2022b).
Our data split technique is a key to avoid the uniform convergence argument with the log covering number that is often used to bound this term in Jin et al.
(2021), which is often large for complex models.
For example, in a two-layer ReLU NTK, the eigenvalues of the induced RKHS has d -polynomial decay (Bietti & Mairal, 2019), thus its log covering number roughly follows, by (Yang et al., 2020, Lemma D1),
[Equation]
for some α ∈ (0 , 1) .
Therefore, for any h ∈ [ H ] , { ϵ kh } k ∈I h is adapted to the filtration {F kh } k ∈I h .
Applying Lemma G. with Z t = ϵ ht ∈ [ − ( H + ψ ) , H + ψ ] , σ {+
+}
=
( H + ψ ) 2 , ρ = λ − 1 , for any δ > 0 , with probability at least 1 − δ/ 3 , for any h ∈
[ H ] ,
[Equation]
Substituting Equation (37) into Equation (36), we have
[Equation]
where the last equation holds due to the definition of the effective dimension.
Combining Equations (38), (35), (33), (31), and (30) via the union bound, with probability at least 1 − Hm − 2 − δ , for any x ∈ S d − 1 and any h ∈ [ H ] ,
[Equation]
where
[Equation]
Combing with Lemma D.3 using the union bound, with probability at least 1 − Hm − 2 − 2 δ , for any x ∈ S d − 1 , and any h ∈ [ H ] ,
[Equation]
where ι 2 , and β are defined in Table 3.
Similarly, it is easy to show that
[Equation]
E. P ROOF OF L EMMA D.
Before proving Lemma D.2, we prove the following intermediate lemmas.
The detailed proofs of these intermediate lemmas are deferred to Section F.
Lemma E.2.
Conditioned on all the randomness except { ξ k,ih } and { ζ ih } , for any i ∈ [ M ] ,
[Equation]
Lemma E.3.
If we set M = log HSAδ / log 1 1 − Φ( − 1) where Φ( · ) is the cumulative distribution function of the standard normal distribution, then with probability at least 1 − δ , for any ( x, h ) ∈
X ×
[ H ] ,
[Equation]
We are now ready to prove Lemma D.2.
Proof of Lemma D.2.
Note that the parameter condition in Equation (20) of Lemma D.2 satisfies
Equation (21) of Lemma D.3, thus given the parameter condition Lemma D.2, Lemma D.3 holds.
For the rest of the proof, we consider under the joint event in which both the inequality of Lemma
D.3 and that of Lemma E.3 hold.
By the union bound, probability that this joint event holds is at least 1 − MHm − 2 − δ .
Thus, for any x ∈ S d − 1 , h ∈ [ H ] , and i ∈
[ M ] , min i
[Equation]
where the first inequality holds due to Lemma D.3, and the second inequality holds due to Lemma
E.3.
Thus, we have
[Equation]
E. P ROOF OF L EMMA D.
In this subsection, we provide a detailed proof of Lemma D.3.
We first provide intermediate lemmas that we use for proving Lemma D.3.
The detailed proofs of these intermediate lemmas are deferred to Section F.
The following lemma bounds the the gradient descent weight of the perturbed loss function around the linear weight counterpart.
Lemma E.4.
Let
[Equation]
where ˜ B 1 , ˜ B 2 , γ h, 1 and γ h, 2 are defined in Table 3 and C g is a constant given in Lemma G.1.
With probability at least 1 − MHm − 2 − δ , for any ( i, j, h ) ∈
[ M ] ×
[ J ] × [ H ] , we have
[Equation]
Similar to Lemma E.
[Equation]
we obtain the following lemma for the gradient descent weights of the nonperturbed loss function.
Lemma E.5.
Let
[Equation]
41) where B 1 , ˜ B 2 , γ h, 1 and γ h, 2 are defined in Table 3 and C g is a constant given in Lemma G.1.
With probability at least 1 − MHm − 2 − δ , for any ( i, j, h ) ∈
[ M ] ×
[ J ] × [ H ] , we have
• ˆ [Equation] ( j ) h ∈
[-( W 0 ; R ) , • ∥ ˆ W ( j ) h − ˆ W linh ∥ 2 ≤ B 1 + ˜ B 
 + λ − 1 (1 − ηλ )-][Equation]
• ∥
[Equation]
We now can prove Lemma D.3.
Proof of Lemma D.3.
Note that Equation (21) implies both Equation (40) of Lemma E.4 and Equation (41) of Lemma E.5, thus both Lemma E.4 and Lemma E.5 holds under Equation (21).
Thus, by the union bound, with probability at least 1 − MHm − 2 − δ , for any ( i, j, h ) ∈
[ M ] ×
[ J ] ×
[ H ] , and x ∈ S
d − 1 ,
[Equation]
where the first inequality holds due to the triangle inequality
[Equation]
the second inequality holds due to Cauchy-Schwarz inequality, Lemma G.
[Equation]
and Lemma E. [Equation]
Similarly, by the union bound, with probability at least 1 − Hm − 2 , for any ( i, j, h ) ∈ [ M ] × [ J ] ×
[ H ] , and x ∈ S
d − 1 ,
[Equation]
where the first inequality holds due to the triangle inequality, the second inequality holds due to
Cauchy-Schwarz inequality, Lemma E.5, and Lemma G.1.
F P ROOFS OF L EMMAS IN S ECTION E
In this section, we provide the detailed proofs of Lemmas in Section E.
F. P ROOF OF L EMMA E.
Proof of Lemma E.1.
As B h ˜ V
h +1 ∈ Q ∗ by Assumption 5.1, where Q ∗ is defined in Section 5, we have
[Equation]
for some c : R d → R d such that sup w ∥ c ( w ) ∥ 2 p 0 ( w ) ≤ B .
The lemma then directly follows from approximation by finite sum (Gao et al., 2019).
F. P ROOF OF L EMMA E.
Proof of Lemma E.2.
Let ¯ W :
= W + ζ ih and ¯ L ih
[Equation] [Equation] W
[Equation]
y [Equation] [Equation] λ ∥ ¯ W ∥ 22 , where ¯ y kh =
r kh + ˜ V
h +
( s kh +1 )
+ ξ k,ih + ⟨ g ( x kh ; W 0 ) , ζ ih ⟩ .
We have ˜ L i,linh ( W ) =
¯ L ih ( ¯ W ) and arg max W ˜ L i,linh ( W ) =
arg max ¯ W ¯ L ih ( ¯ W ) − ζ ih as both ˜ L i,linh ( W ) and ¯ L ih are convex.
Using the regularized least-squares solution,
[Equation]
[Equation]
Thus, we have
[Equation]
By direct computation, it is easy to see that
[Equation]
F.  P ROOF OF L EMMA E.
In this subsection, we provide a proof for E.3.
We first provide a bound for the perturbed noises used in Algorithm 1 in the following lemma.
Lemma F.1.
There exist absolute constants c 1 , c 2 > 0 such that for any δ > 0 , event E ( δ ) holds with probability at least 1 − δ , for any ( k, h, i ) ∈
[ K ] ×
[ H ] ×
[ M ] ,
[Equation]
Proof of Lemma F.1.
It directly follows from the Gaussian concentration inequality in Lemma G. and the union bound.
We now can prove Lemma E.3.
Proof of Lemma E.3.
By Lemma E.2,
[Equation]
Using the anti-concentration of Gaussian distribution, for any x = ( s, a ) ∈ S × A and any i ∈
[ M ] ,
[Equation]
As { ˜ W i,lin h } i ∈ [ M ] are independent, using the union bound, with probability at least 1 − SAH (1 − Φ( − 1))
M , for any x
= ( s, a ) ∈ S × A , and h ∈
[ H ] ,
[Equation]
Setting δ = SAH (1 − Φ( − 1))
M completes the proof.
F. P ROOF OF L EMMA E.
In this subsection, we provide a detailed proof of Lemma E.4.
We first prove the following intermediate lemma whose proof is deferred to Subsection F.5.
Lemma F.2.
Let
[Equation]
and additionally let
[Equation]
Then with probability at least 1 − MHm − 2 − δ , for any ( i, j, h ) ∈
[ M ] ×
[ J ] × [ H ] , if ˜ W i, ( j ) h ∈
B ( W 0 ; R ) for any j ′ ∈
[ j ] , then
[Equation]
We now can prove Lemma E.4.
Proof of Lemma E.4.
To simplify the notations, we define
[Equation]
The gradient descent update rule for ˜ W i, ( j ) h in Equation (9) can be written as:
[Equation]
The auxiliary updates in Equation (15) can be written as:
[Equation]
Step 1: Proving ˜ W i, ( j ) h ∈ B ( W 0 ; R ) for all j .
In the first step, we prove by induction that with probability at least 1 − MHm − 2 − δ , for any ( i, j, h ) ∈
[ M ] ×
[ J ] × [ H ] , we have
[Equation]
In the rest of the proof, we consider under the event that Lemma F.2 holds.
Note that the condition in Lemma E.4 satisfies that of Lemma F.2 and under the above event of Lemma F.2, Lemma G.1 and
Lemma F.1 both hold.
It is trivial that
[Equation]
For any fix j ≥ 0 , we assume that
[Equation]
We will prove that ˜ W i, ( j +1) h ∈ B ( W
[Equation]
R ) .
We have ∥ ∆ j +1 ∥
[Equation]
We bound I 1 , I
and I 3 separately.
Bounding I 1 .
For bounding I 1 ,
[Equation]
where the first inequality holds due to the spectral norm inequality [Equation] the second inequality holds due to
[Equation]
where the first inequality holds due to that H 0 ⪯ ∥ H
∥ 2 I ⪯ ∥ G 0 ∥
I , the second inequality holds due to that ∥ G 0 ∥ 2 ≤ √ KC g due to Lemma G.1, and the last inequality holds due to the choice of η in Equation (40).
Bounding I 2 .
For bounding I 2 ,
[Equation]
where the first inequality holds due to Cauchy-Schwarz inequality, the second inequality holds due to the induction assumption in Equation (42) and Lemma G.1, and the third inequality holds due to
Lemma F.2 and the induction assumption in Equation (42).
Bounding I 3 .
For bounding I 3 ,
[Equation]
where the first inequality holds due to Cauchy-Schwarz inequality and due to that ∥ G 0 ∥ 2 ≤ √ K ′ C g and the second inequality holds due to the induction assumption in Equation (42) and Lemma G.1.
Combining the bounds of I 1 , I 2 , I 3 above, we have
[Equation]
Recursively applying the inequality above for all j , we have
[Equation]
where the second inequality holds due the choice specified in Equation (40).
We also have
[Equation]
[Equation]
where the first inequality holds due the the definition of ˜ L i,linh ( ˜ W
i,lin, ( j +1) h ) , the second inequality holds due to the monotonicity of ˜ L i,linh ( W ) on the gradient descent updates { ˜ W
i,lin, ( j ′ ) h } j ′ for the squared loss on a linear model, the third equality holds due to ⟨ g ( x kh ; W 0 ) , W 0 ⟩ = 0 from the symmetric initialization scheme, and the last inequality holds due to Lemma F.1.
Thus, we have ∥
[Equation]
where the first inequality holds due to Cauchy-Schwarz inequality, the second inequality holds due to Equation (44) and Lemma F.1, and the last inequality holds due to the choice specified in Equation (40).
Combining Equation (43) and Equation (45), we have
[Equation]
where the first inequality holds due to the triangle inequality.
Step 2: Bounding ∥ ˜ W
i, ( j ) h − ˜ W i,linh ∥ 2 .
By the standard result of gradient descent on ridge linear regression, ˜ W
i, ( j ) h converges to ˜ W
i,linh with the convergence rate,
[Equation]
Thus, for any j , we have
[Equation]
where the first inequality holds due to the triangle inequality, the second inequality holds due to
Equation (43) and Equation (46).
F. P ROOF OF L EMMA F.
Proof of Lemma F.2.
We bound this term following the proof flow of (Zhou et al., 2020, Lemma C.3) with modifications for different neural parameterization and noisy targets.
Suppose that for some fixed j ,
[Equation]
Let us define
[Equation]
× K ′ ,
[Equation]
To further simplify the notations in this proof, we drop i, h in ˜ L ih ( W ) defined in Equation (8) to write ˜ L ih ( W ) as ˜ L ( W ) and write W j = W i, ( j ) h , where
[Equation]
Suppose that W ∈ B ( W 0 ; R ) .
By that ∥ · ∥ 22 is 1 -smooth, ˜
[Equation]
I
[Equation]
[Equation]
For bounding I 1 ,
[Equation]
) where the first inequality holds due to Cauchy-Schwarz inequality, W ∈ B ( W 0 ; R ) and Lemma G.1.
Substituting Equation (49) into Equation (48) with W ′ =
W − η ∇ ˜ L ( W ) ,
[Equation]
By the 1 -strong convexity of ∥ · ∥
[Equation]
for any W ′ , ˜
[Equation]
where the last inequality holds due to Cauchy-Schwarz inequality.
Substituting Equation (51) into Equation (50), for any W ′ ,
˜ L ( W − η ∇ ˜ L ( W ))
− ˜ L ( W ) ≤ 2 λη (1 − ( KC 2 g + λ/ 2) η ) (cid:124) (cid:123)(cid:122) (cid:125) α (cid:16) ˜ L ( W ′ ) − ˜ L ( W ) + ∥ f ( W ) − ˜ y
∥
∥ e ( W ′ , W ) ∥ 2 (cid:17) + ∥ f ( W ) − ˜ y
{+∥ 
 ∥ e ( W − η ∇ ˜ L ( W ) , W ) ∥ 2 + ∥ e ( W − η ∇ ˜ L ( W ) ,+} W ) ∥ {+
+}
[Equation]
where the second inequality holds due to Cauchy-Schwarz inequality for any γ 1 , γ 2 > 0 , and the third inequality holds due to ∥ f ( W ) − ˜ y
∥ 22 ≤ 2˜ L ( W ) .
Rearranging terms in Equation (52) and setting W = W j , W ′ =
[Equation]
0 , γ
[Equation]
[Equation] γ
[Equation]
α
[Equation]
53) where e := C g R 4 / 3 m − 1 / 6 √ log m , the last inequality holds due to Equation (47) and Lemma G.1.
Applying Equation (53), we have
[Equation]
Rearranging the above inequality,
[Equation]
where the last inequality holds due to the choice of η .
Finally, we have
[Equation]
and ˜ L ( W 0 )
= 12 ∥
˜ y
∥
22 + λ 2 ∥ ζ ih ∥ 22 ≤ K ′ 2 ( H + ψ + γ h, 1 )
+ λ 2 γ 2 h, 2 due to Lemma F.1.
G S UPPORT L EMMAS
Lemma G.1.
Let m =
Ω (cid:16) d 3 / 2 R − 1 log 3 / 2 ( √ m/R ) (cid:17) and R = O (cid:0) m 1 / 2 log − 3 m (cid:1) .
With probability at least 1 − e − Ω(log 2 m ) ≥ 1 − m − 2 with respect to the random initialization, it holds for any
W, W ′ ∈ B ( W 0 ; R ) and x ∈ S d − 1 that
[Equation]
where C g =
O
(1) is a constant independent of d and m .
Moreover, without loss of generality, we assume C g ≤ 1 .
Proof of Lemma G.1.
Due to (Yang et al., 2020, Lemma C.2) and (Cai et al., 2019, Lemma F.1, F.2), we have the first two inequalities and the following:
[Equation]
For any W, W ′ ∈ B ( W 0 ; R ) ,
[Equation]
Thus,
[Equation]
Lemma G.2 ((Arora et al., 2019, Theorem 3)) .
If m = Ω( ϵ − 4 log(1 /δ )) , then for any x, x ′ ∈ X ⊂ S d − 1 , with probability at least 1 − δ ,
[Equation]
Lemma G.3.
Let X ∼ N (0 , a Λ − 1 ) be a d -dimensional normal variable where a is a scalar.
There exists an absolute constant c > 0 such that for any δ > 0 , with probability at least 1 − δ ,
[Equation]
For d = 1 , c = √ 2 .
Lemma G.4 (A variant of Hoeffding-Azuma inequality) .
Suppose { Z k } ∞ k =0 is a real-valued stochastic process with corresponding filtration {F k } ∞ k =0 , i.e. ∀ k , Z k is F k -measurable.
Suppose that for any k , E [ | Z k | ] < ∞ and | Z k − E
[ Z k |F k − 1 ] | ≤ c k almost surely.
Then for all positive n and t , we have:
[Equation]
Lemma G.5 ((Chowdhury & Gopalan, 2017, Theorem 1)) .
Let H be an RKHS defined over X ⊆ R d .
Let { x t } ∞ t =1 be a discrete time stochastic process adapted to filtration {F t } ∞ t =0 .
Let { Z k } ∞ k
= be a real-valued stochastic process such that Z k ∈ F k , and Z k is zero-mean and σ -sub Gaussian conditioned on F k − 1 .
Let E k
= ( Z 1 , . .
, Z k − 1 ) T ∈ R k − 1 and K k be the Gram matrix of H defined on { x t } t ≤ k − 1 .
For any ρ > 0 and δ ∈ (0 , 1) , with probability at least 1 − δ ,
[Equation]
Lemma G.6.
For any matrices A and B where A is invertible,
[Equation]
Lemma G.7.
For any invertible matrices A, B ,
[Equation]
Proof of Lemma G.7.
We have:
[Equation]
Lemma G.8 (Freedman’s inequality (Tropp, 2011)) .
Let { X k } nk
=
1 be a real-valued martingale difference sequence with the corresponding filtration {F k } nk =1 , i.e. X k is F k -measurable and E
[ X k |F k − 1 ] = 0 .
Suppose for any k , | X k | ≤ M almost surely and define V :=
(cid:80) nk =
[Equation] E
[Equation]
X 2 k |F k − 1 (cid:3) .
For any a, b > 0 , we have:
[Equation]
In an alternative form, for any t > 0 , we have:
[Equation]
Lemma G.9 (Improved online-to-batch argument Nguyen-Tang et al.
(2022c)) . Let { X k } be any real-valued stochastic process adapted to the filtration {F k } , i.e. X k is F k -measurable.
Suppose that for any k , X k ∈ [0 , H ] almost surely for some H > 0 .
For any K > 0 , with probability at least 1 − δ , we have:
[Equation]
Proof of Lemma G.9 .
Let Z k
=
X
k − E
[ X k |F k − 1 ] and f ( K ) =
(cid:80) Kk =1 E [ X k |F k − 1 ] .
We have Z k is a real-valued difference martingale with the corresponding filtration {F k } and that
[Equation]
Note that | Z k | ≤ H and f ( K ) ∈ [0 , KH ] and let m = log 2 ( KH ) .
Also note that f ( K ) =
(cid:80)
Kk
[Equation] − (cid:80) Kk =1 Z k ≥ − (cid:80)
Kk =1 Z k .
Thus if (cid:80) Kk =1 Z k ≤ − 1 , we have f ( K ) ≥ 1 .
For any t > 0 , leveraging the peeling technique (Bartlett et al., 2005), we have:
[Equation]
where the first equation is by that (cid:80)
Kk {+=1 Z k ≤ − 2 Ht 3 − (cid:112) 
+}
Hf ( K ) t − 1 ≤ − 1 thus f ( K ) ≥ 1 , the second inequality is by that V ≤
Hf ( K ) , and the last inequality is by Lemma G.8.
Thus, with probability at least 1 − me − t , we have:
[Equation]
The above inequality implies that f ( K ) ≤ 2 (cid:80)
Kk =1 X k + 4 Ht/ 3 + 2 + 4 Ht , due to the simple inequality: if x ≤ a √ x + b , x ≤ a 2 + 2 b .
Then setting t = log( m/δ ) completes the proof.
H B ASELINE ALGORITHMS
For completeness, we give the definition of linear MDPs as follows.
Definition 3 (Linear MDPs (Yang & Wang, 2019; Jin et al., 2020)) .
An MDP has a linear structure if for any ( s, a, s ′ , h ) , we have:
[Equation]
where ϕ : S ×
A → R d lin is a known feature map, θ
h ∈ R
d lin is an unknown vector, and µ h : S → R d lin are unknown signed measures.
We aslo give the details of the baseline algorithms: LinLCB in Algorithm 3, LinGreedy in Algorithm 4, LinPER in Algorithm 5, NeuraLCB in Algorithm 6 and NeuralGreedy in Algorithm 7.
For simplicity, we do not use data split in these algorithms presented here.
← β ∥ h ( Σ − θ ⟩ − b
H − h + h ← arg max π h ⟨ ˆ Q h
1: Input : Offline data D = { ( s kh , a kh , r kh ) } k ∈
[ K ] h ∈ [ H ] , perturbed variances { σ h } h ∈ [ H ] , number of bootstraps M , regularization parameter λ .
2: Initialize ˜ V
H +1 ( · ) ←3: for h = H, . . .
, 1 do 4: Λ h ← (cid:80) Kk =1 ϕ ( s kh , a kh ) ϕ
( s kh , a kh ) T + λI 5: for i = 1 , . . .
, M do 6:
Sample { ξ τ,ih } τ ∈
[ K ] ∼ N (0 , σ 2 h ) and ζ ih = { ζ j,ih } j ∈ [ d ] ∼ N (0 , σ 2 h I d ) 7:
Solve the perturbed regularized least-squares regression: ˜ θ ih ← arg max θ ∈ R d
K (cid:88) k
= (cid:16) ⟨ ϕ ( s kh , a kh ) , θ ⟩ − ( r kh + ˜ V h
+
( s kh +1 )
+ ξ k,ih ) (cid:17) 2 + λ ∥ θ + ζ ih ∥ 22 , 8: end for 9:
Compute ˜ Q h ( · , · ) ← min { min i ∈ [ M ] ⟨ ϕ ( · , · ) , ˜ θ ih ⟩ , H − h + 1 } + 10: ˜
π h ← arg max π h ⟨ ˜ Q h , π h ⟩ and ˜ V
h ← ⟨ ˜ Q h , ˜ π h ⟩ 11: end for 12: Output : ˜ π
= { ˜ π h } h ∈
[ H ]
1: Input : Offline data D = { ( s kh , a kh , r kh ) } k ∈ [ K ] h ∈ [ H ] , neural networks F = { f ( · , · ; W ) : W ∈ W} ⊂ {X → R } do 4: ˆ W h ← GradientDescent ( λ, η, J, { ( s kh ,
a kh , r kh ) } k ∈ [ K ] , 0 , W 0 ) ( 2) 5: Λ h = λI + (cid:80)
Kk =1 g ( s kh , a kh ; ˆ W h ) g ( x kh ; ˆ W h ) T 6: Compute ˆ Q h ( · , · ) ← min { f ( · , · ; ˆ W h ) − β ∥ g ( · , · ; ˆ W h ) ∥ Λ − 1 h , H − h + 1 } + 7: ˆ π h ← arg max π h ⟨ ˆ Q h , π h ⟩ and ˆ V h ← ⟨ ˆ Q h , ˆ π h ⟩ end Compute
← ←