O PTIMISM IN R EINFORCEMENT L EARNING WITH G ENERALIZED L INEAR F UNCTION A PPROXIMATION
A BSTRACT
We design a new provably efﬁcient algorithm for episodic reinforcement learning with generalized linear function approximation.
We analyze the algorithm under a new expressivity assumption that we call “optimistic closure ” which is strictly weaker than assumptions from prior analyses for the linear setting.
With optimistic closure, we prove that our algorithm enjoys a regret bound of (cid:101)
O (cid:16) H √ d 3 T (cid:17) where H is the horizon, d is the dimensionality of the state-action features and T is the number of episodes.
This is the ﬁrst statistically and computationally efﬁcient algorithm for reinforcement learning with generalized linear functions
I NTRODUCTION
We study episodic reinforcement learning problems with inﬁnitely large state spaces, where the agent must use function approximation to generalize across states while simultaneously engaging in strategic exploration.
Such problems form the core of modern empirical/deep-RL, but relatively little work focuses on exploration, and even fewer algorithms enjoy strong sample efﬁciency guarantees.
On the theoretical side, classical sample efﬁciency results from the early 00s focus on “tabular” environments with small ﬁnite state spaces (Kearns & Singh, 2002; Brafman & Tennenholtz, 2002;
Strehl et al., 2006), but as these methods scale with the number of states, they do not address problems with inﬁnite or large state spaces.
While this classical work has inspired practically effective approaches for large state spaces (Bellemare et al., 2016;
Osband et al., 2016; Tang et al., 2017), these methods do not enjoy sample efﬁciency guarantees.
More recent theoretical progress has produced provably sample efﬁcient algorithms for complex environments where function approximation is required, but these algorithms are relatively impractical (Krishnamurthy et al., 2016; Jiang et al., 2017).
In particular, these methods are computationally inefﬁcient or rely crucially on strong dynamics assumptions (Du et al., 2019b).
In this paper, with an eye toward practicality, we study a simple variation of Q-learning, where we approximate the optimal Q-function with a generalized linear model.
The algorithm is appealingly simple: collect a trajectory by following the greedy policy corresponding to the current model, perform a dynamic programming back-up to update the model, and repeat.
The key difference over traditional Q-learning-like algorithms is in the dynamic programming step.
Here we ensure that the updated model is optimistic in the sense that it always overestimates the optimal Q-function.
This optimism is essential for our guarantees.
Optimism in the face of uncertainty is a well-understood and powerful algorithmic principle in shorthorizon (e.g,.
bandit) problems, as well as in tabular reinforcement learning (Azar et al., 2017; Dann et al., 2017; Jin et al., 2018).
With linear function approximation, Yang & Wang (2019) and Jin et al.
(2019) show that the optimism principle can also yield provably sample-efﬁcient algorithms, when the environment dynamics satisfy certain linearity properties.
Their assumptions are always satisﬁed in tabular problems, but are somewhat unnatural in settings where function approximation is required.
Moreover as these assumptions are directly on the dynamics, it is unclear how their analysis can accommodate other forms of function approximation, including generalized linear models.
In the present paper, we replace explicit dynamics assumptions with expressivity assumptions on the function approximator, and, by analyzing a similar algorithm to Jin et al.
(2019), we show that the optimism principle succeeds under these strictly weaker assumptions.
1 More importantly, the relaxed assumption facilitates moving beyond linear models, and we demonstrate this by providing the ﬁrst practical and provably efﬁcient RL algorithm with generalized linear function approximation.
The paper is organized as follows:
In Section 2 we formalize our setting, introduce the optimistic closure assumption, and discuss related assumptions in the literature.
In Section 3 we study optimistic closure in detail and verify that it is strictly weaker than the recently proposed Linear MDP assumption.
Our main algorithm and results are presented in Section 4, with the main proof in Section A.
We close with some ﬁnal remarks and future directions in Section 5.
P RELIMINARIES
We consider episodic reinforcement learning in a ﬁnite-horizon markov decision process (MDP) with possibly inﬁnitely large state space S , ﬁnite action space A , initial distribution µ ∈ ∆( S ) , transition operator P : S ×
A → ∆( S ) , reward function R : S ×
A → ∆([0 , 1]) and horizon H .
The agent interacts with the MDP in episodes and, in each episode, a trajectory ( s 1 , a 1 , r 1 , s 2 , a 2 , r 2 , . . .
, s H , a H , r H ) is generated where s 1 ∼ µ , for h > 1 we have s h ∼ P ( · | s h − 1 , a h − 1 ) , r h ∼ R ( s h , a h ) , and actions a 1: H are chosen by the agent.
For normalization, we assume that (cid:80)
Hh =1 r h ∈ [0 , 1] almost surely.
A (deterministic, nonstationary) policy π = ( π 1 , · · · , π H ) consists of H mappings π h : S → A , where π h ( s h ) denotes the action to be taken at time point h
if at state s h ∈ S
The value function for a policy π is a collection of functions ( V π 1 , . .
. , V πH )
where V πh : S → R is the expected future reward the policy collects if it starts in a particular state at time point h .
Formally,
[Equation]
The value for a policy π is simply V π (cid:44) E s 1 ∼ µ [ V π 1 ( s 1 )] , and the optimal value is V (cid:63) (cid:44) max π V π , where the maximization is over all nonstationary policies.
The typical goal is to ﬁnd an approximately optimal policy, and in this paper, we measure performance by the regret accumulated over T episodes,
[Equation]
Here r h,t is the reward collected by the agent at time point h in the t th episode.
We seek algorithms with regret that is sublinear in T , which demonstrates the agent’s ability to act near-optimally over the long run.
2. Q- VALUES AND FUNCTION APPROXIMATION
For any policy π , the state-action value function, or the Q -function is a sequence of mappings
Q π =
( Q π 1 , . . .
, Q πH ) where Q πh : S ×
A → R is deﬁned as
[Equation]
The optimal Q -function is Q (cid:63)h (cid:44) Q π (cid:63) h where π (cid:63) (cid:44) argmax
π V π is the optimal policy.
In the value-based function approximation setting, we use a function class G to model Q (cid:63) .
In this paper, we always take G to be a class of generalized linear models (GLMs), deﬁned as follows: Let d ∈ N be a dimensionality parameter and let B d (cid:44) (cid:8) x ∈ R d : (cid:107) x (cid:107) 2 ≤ 1 (cid:9) be the (cid:96) 2 ball in R d .
Deﬁnition 1.
For a known feature map φ : S ×
A → B d and a known link function f : [ − 1 , 1] (cid:55)→ [ − 1 , 1] the class of generalized linear models is G (cid:44) { ( s, a ) (cid:55)→ f
( (cid:104) φ ( s, a ) , θ (cid:105) ) :
θ ∈ B d } .
As is standard in the literature (Filippi et al., 2010; Li et al., 2017), we assume the link function satisﬁes certain regularity conditions.
Assumption 1.
f ( · ) is either monotonically increasing or decreasing.
Furthermore, there exist absolute constants 0 < κ < K < ∞ and M < ∞ such that κ ≤ | f (cid:48)
( z ) | ≤ K and | f (cid:48)(cid:48)
( z ) | ≤ M for all | z | ≤ 1 .
For intuition, two example link functions are the identity map f ( z ) =
z and the logistic map f ( z )
= 1 / (1 + e − z ) with bounded z .
It is easy to verify that both of these maps satisfy Assumption 1.
2. E XPRESSIVITY ASSUMPTIONS : REALIZABILITY AND OPTIMISTIC CLOSURE
To obtain sample complexity guarantees that scale polynomially with problem parameters in the function approximation setting, it is necessary to posit expressivity assumptions on the function class G (Krishnamurthy et al., 2016; Du et al., 2019a).
The weakest such condition is realizability , which posits that the optimal Q function is in G , or at least well-approximated by G .
Realizability alone sufﬁces for provably efﬁcient algorithms in the “contextual bandits” setting
where H = 1 (Li et al., 2017; Filippi et al., 2010; Abbasi-Yadkori et al., 2011), but it does not seem to be sufﬁcient when H > 1 .
Indeed in these settings it is common to make stronger expressivity assumptions
(Chen & Jiang, 2019; Yang & Wang, 2019; Jin et al., 2019).
Following these works, our main assumption is a closure property of the Bellman update operator
T h .
This operator has type T h : ( S ×
A → R )
→ ( S × A → R ) and is deﬁned for all s ∈ S , a ∈ A as
[Equation]
The Bellman update operator for time point H is simply T H ( Q )( s, a ) (cid:44) E
[ r H | s H = s, a H = a ] , which is degenerate.
To state the assumption, we must ﬁrst deﬁne the enlarged function class G up .
For a d × d matrix A , A (cid:23) 0 denotes that A is positive semi-deﬁnite.
For a positive semi-deﬁnite matrix A , (cid:107) A (cid:107) op is the matrix operator norm, which is just the largest eigenvalue, and (cid:107) x (cid:107) A (cid:44) √ x
(cid:62)
Ax is the matrix Mahalanobis seminorm.
For a ﬁxed constant Γ ∈ R + that we will set to be polynomial in d and log( T ) , deﬁne
[Equation]
Here we use a ∧ b (cid:44) min { a, b } .
The class G up contains G in addition to all possible upper conﬁdence bounds that arise from solving least squares regression problems using the class G .
We now state our main expressivity assumption, which we call optimistic closure .
Assumption 2 (Optimistic closure) .
For any 1 ≤ h < H and g ∈ G up , we have T h ( g ) ∈ G .
In words, when we perform a Bellman backup on any upper conﬁdence bound function for time point h + 1 , we obtain a generalized linear function at time h .
While this property seems quite strong, we note that a similar notion is mentioned informally in Jin et al.
and that related closure-type assumptions are common in the literature (see Section 2.3 for detailed discussion).
More importantly, we will prove in Section 3 that optimistic closure is actually strictly weaker than previous assumptions used in our RL setting where exploration is required.
Before turning to these discussions, we mention two basic properties of optimistic closure.
Fact 1 (Optimistic closure and realizability) .
Optimistic closure implies that Q (cid:63) ∈ G (realizability).
Proof.
We will solve for Q (cid:63) via dynamic programming, starting from time point H .
In this case, the Bellman update operator is degenerate, and we start by observing that T H ( g ) ≡ Q (cid:63)H for all g .
Consequently we have Q (cid:63)H ∈ G .
Next, inductively we assume that we have Q (cid:63)h +1 ∈ G , which implies that Q (cid:63)h +1 ∈ G up as we may take the same parameter θ and set A ≡ 0 .
Then, by the standard
Bellman ﬁxed-point characterization, we know
that Q (cid:63)h = T h ( Q (cid:63)h +1 ) , at which point Assumptionyields that Q (cid:63) h ∈ G .
Fact 2 (Optimistic closure in tabular settings) .
If S is ﬁnite and φ ( s, a ) =
e s,a is the standard-basis feature map, then under Assumption 1 we have optimistic closure.
Proof.
We simply verify that G contains all mappings from ( s, a ) (cid:55)→ [0 , 1] , at which point the result is immediate.
To see why, observe that via Assumption 1 we know that f is invertible (it is monotonic with derivative bounded from above and below).
Then, note that any function ( s, a ) (cid:55)→ [0 , 1] can be written as a vector v ∈ [0 , 1] |S|×|A| .
For such a vector v , if we deﬁne θ s,a (cid:44) f − 1 ( v s,a ) we have that f ( (cid:104) e s,a , θ (cid:105) )
= v s,a .
Hence G contains all functions, so we trivially have optimistic closure.
2. R ELATED WORK
The majority of the theoretical results for reinforcement learning focus on the tabular setting where the state space is ﬁnite and sample complexities scaling polynomially with |S| are tolerable (Kearns & Singh, 2002; Brafman & Tennenholtz, 2002; Strehl et al., 2006).
Indeed, by now there are a number of algorithms that achieve strong guarantees in this setting (Dann et al., 2017; Azar et al., 2017;
Jin et al., 2018; Simchowitz & Jamieson, 2019).
Via Fact 2, our results apply to this setting, and indeed our algorithm can be viewed as a generalization of the canonical tabular algorithm (Azar et al., 2017; Dann et al., 2017; Simchowitz & Jamieson, 2019) to the function approximation setting.
Turning to the function approximation setting, several other results concern function approximation in settings where exploration is not an issue, including the inﬁnite-data regime (Munos, 2003; Farahmand et al., 2010) and the “batch RL” setting where the agent does not control the data-collection process (Munos & Szepesv ´ ari, 2008; Antos et al., 2008; Chen & Jiang, 2019).
While the details differ, all of these results require that the function class satisfy some form of (approximate) closure with respect to the Bellman operator.
As an example, one assumption is that T ( g ) ∈ G for all g ∈ G , with an appropriately deﬁned approximate variant (Chen & Jiang, 2019).
These results therefore provide motivation for our optimistic closure assumption.
While optimistic closure is stronger than the assumptions in these works, we emphasize that we are also addressing exploration, so our setting is also signiﬁcantly more challenging.
A recent line of work studies function approximation in settings where the agent must explore the environment (Krishnamurthy et al., 2016; Jiang et al., 2017; Du et al., 2019b).
The algorithms developed here can accommodate function classes beyond generalized linear models, but they are still relatively impractical and the more practical ones require strong dynamics assumptions (Du et al., 2019b).
In contrast, our algorithm is straightforward to implement and does not require any explicit dynamics assumption.
As such, we view these results as complementary to our own.
Most closely related to our work are the recent results of Yang & Wang (2019) and Jin et al.
Both papers study MDPs with certain linear dynamics assumptions (what they call the Linear MDP assumption) and use linear function approximation to obtain provably efﬁcient algorithms.
Jin et al.
(2019) hint at optimistic closure as a weakening of their Linear MDP assumption and remark that their guarantees continues to hold under this weaker assumption.
One of our contributions is to formalize this remark.
Indeed, our algorithm is almost identical to theirs.
However we emphasize that optimistic closure is strictly weaker than their Linear MDP assumption, which in turn is strictly weaker than the assumption of Yang & Wang (2019).
Further, and perhaps more importantly, by avoiding explicit dynamics assumptions, we enable approximation with GLMs, which are incompatible with the Linear MDP structure.
Hence, the present paper can be seen as a signiﬁcant generalization of these recent results.
Since the initial version of this paper appeared, several other works have studied linear function approximation in reinforcement learning.
A number of papers (Cai et al., 2019; Ayoub et al., 2020;
Modi et al., 2020; Zhou et al., 2020) study an incomparable class of dynamics models that permit linear function approximation.
Others study weakenings of the Linear MDP assumptions.
In particular, Agarwal et al.
(2020) only require small transfer error for linear regression, which formalizes out-of-distribution generalization and is always zero in Linear MDPs.
Zanette et al.
(2020a) only require that the Bellman operator is closed with respect to linear functions, which is considerably weaker than our optimistic closure assumption.
However, their algorithm is not computationally efﬁcient.
Computational efﬁciency is addressed in Zanette et al.
(2020b) in the reward-free setting with reachability assumptions.
As we do not require reachability assumptions, this latter result is incomparable to ours.
None of these results considers generalized linear models.
O N OPTIMISTIC CLOSURE
For a more detailed comparison to the recent work of Yang & Wang (2019) and Jin et al.
(2019), we deﬁne the linear MDP model studied in the latter work.
Deﬁnition 2.
An MDP is said to be a linear MDP if there exist known feature map ψ : S ×A → R d , unknown signed measures µ : S → R d , and an unknown vector η ∈ R d such that (1) P ( s (cid:48) | s, a ) = (cid:104) ψ ( s, a ) , µ ( s (cid:48) )
(cid:105) holds for all states s, s (cid:48) and actions a , and (2) E [ r | s, a ] = (cid:104) ψ ( s, a ) , η (cid:105) .
Linear MDPs are studied by Jin et al.
(2019), who establish a √ T -type regret bound for an optimistic algorithm.
This assumption already subsumes that of Yang & Wang (2019), and related assumptions also appear elsewhere in the literature (Bradtke & Barto, 1996; Melo & Ribeiro, 2007; Zanette et al., 2019).
In this section, we show that optimistic closure (Assumption 2) is strictly weaker than assuming the environment is a linear MDP.
Proposition 1.
If an MDP is linear then Assumption 2 holds with G =
{ ( s, a ) (cid:55)→ (cid:104) w, ψ ( s, a ) (cid:105) :
w ∈ B d } .
Proof.
The result is implicit in Jin et al.
, and we include the proof for completeness.
For any function g , observe that owing to the linear MDP property
T
[Equation]
which is clearly a linear function in ψ ( s, a ) .
Hence for any function g , which trivially includes the optimistic functions, we have T h ( g ) ∈ G .
Thus the linear MDP assumption is stronger than Assumption 2.
Next, we show that it is strictly stronger.
Proposition 2.
There exists an MDP with H = 2 , d = 2 , |A| = 2 and |S| = ∞ such that Assumption 2 is satisﬁed, but the MDP is not a linear MDP.
Thus we have that optimistic closure is strictly weaker than the linear MDP assumption from Jin et al.
Thus, our results strictly generalize theirs.
Proof.
Fix the link function to be f ( z ) =
z .
We ﬁrst construct the MDP.
Set the action space
A = { a 1 , a 2 } .
We use e i to denote the i th standard basis element, and let x = (0 . 1 / Γ , 0 .
1 / Γ) be a ﬁxed vector where Γ appears in the construction of G up .
Recall that s 1 is the ﬁrst state in each trajectory.
In our example, for all a ∈ A , φ ( s 1 , a ) is sampled uniformly at random from the set { αe 1 + (1 − α )
e 2 : α ∈ [0 , 1] } .
The transition rule is deterministic and given by:
[Equation]
Moreover, for the reward function, R ( s 1 , a ) = 0 and R ( s 2 , a ) = 0 .
1 α/ Γ .
We ﬁrst show that the Linear MDP property does not hold for the constructed MDP and the given feature map φ .
Let s (1)1 be the state with φ ( s (1)1 , a ) =
e 2 and s (2)1 be the state with φ ( s (2)1 , a ) =
e 1 .
Notice that we deterministically transition from s (1)1 to a state s (1)2 with φ ( s (1)2 , a ) = 0 , and we deterministically transition from s (2)1 to a state s (2)2 with φ ( s (2)2 , a ) = x , which already ﬁxes the whole transition operator under the linear MDP assumption.
Thus, under the linear MDP assumption, we must therefore have a randomized transition for any state s 1 with φ ( s 1 , a ) =
αe 1 + (1 − α ) ewhere α ∈ (0 , 1) .
This contradicts the fact that our constructed MDP has deterministic transitions everywhere, so the linear MDP cannot hold.
We next show that Assumption 2 holds.
Consider an arbitrary optimistic Q estimate of the form g ( z ) =
min { 1 , z (cid:62) θ + γ √ z (cid:62) Az } ∈ G up .
Notice that for x = (0 . 1 / Γ , 0 .
1 / Γ)
, we always have that φ ( s h τ a h τ y h τ (cid:44) r h,τ + max a (cid:48) ∈A ¯ Q h
+1 ,t ( s h
+1 ,τ , a (cid:48)
ˆ θ h,t (cid:44) argmin (cid:107) θ (cid:107) 2 ≤(cid:88)
τ ≤ t ( y h,τ − f ( (cid:104) x h,τ , θ (cid:105) )) 2 ; (1) 9:
Compute Λ h,t (cid:44) (cid:80) τ ≤ t x h,τ x (cid:62) h,τ
+ I ; 10:
Construct ¯ Q h,t ( s, a ) (cid:44) min (cid:110) 1 , f ( φ ( s, a ) (cid:62) ˆ θ h,t ) + γ (cid:107) φ ( s, a ) (cid:107) Λ − 1 h,t (cid:111) ; 11: 12: end for x [Equation] θ + γ √
[Equation]
Ax ≤ 1 for any θ ∈ B d and A with (cid:107) A (cid:107) op ≤ 1 .
Moreover, for all s 2 , i.e., the second state in the trajectory, we always have φ ( s 2 , a ) =
αx for some α ∈ [0 , 1] .
Hence we can ignore the ﬁrst term in the minimum, and, by direct calculation, we have that when φ ( s, a ) =
αe 1 +(1 − α )
e 2 :
[Equation]
Hence we can write T 1 ( g )
=
(cid:104) φ ( s, a ) , ( c 0 , 0) (cid:105) , which veriﬁes Assumption 2.
A LGORITHM AND MAIN RESULT
We now turn to presenting our algorithm and main results.
We study a least-squares dynamic programming style algorithm that we call LSVI-UCB, with pseudocode presented in Algorithm 1.
The algorithm is nearly identical to the algorithm proposed by Jin et al.
with the same name.
A similar algorithmic template has also been extensively studied in the tabular setting Azar et al.
(2017); Dann et al.
; Simchowitz & Jamieson (2019), albeit with slightly different conﬁdence bounds.
As our algorithm applies to all of these settings, it should be considered as a generalization.
The algorithm uses dynamic programming to maintain optimistic Q function estimates { ¯ Q h,t } h ≤ H,t ≤ T for each time point h and each episode t .
In the t th episode, we use the previously computed estimates to deﬁne the greedy policy ˆ π h,t ( · ) (cid:44) argmax a ∈A ¯ Q h,t − 1 ( · , a ) , which we use to take actions for the episode.
Then, with all of the trajectories collected so far, we perform a dynamic programming update, where the main per-step optimization problem is (1).
Starting from time point H , we update our Q function estimates by solving constrained least squares problems using our class of GLMs.
At time point H , the covariates are { φ ( s H,τ , a H,τ ) } τ ≤ t , and the regression targets are simply the immediate rewards { r H,τ } τ ≤ t .
For time points h < H , the covariates are deﬁned similarly as { φ ( s h,τ , a h,τ ) } τ ≤ t but the regression targets are deﬁned by inﬂating the learned
Q function for time point h + 1 by an optimism bonus .
In detail, the least squares problem for time point h + 1 yields a parameter ˆ θ h +1 ,t and we also form the second moment matrix Λ h
+1 ,t of all the covariates at time h
+ 1 that we have seen so far.
Using these, we deﬁne the optimistic Q function ¯ Q h +1 ,t ( s, a ) (cid:44) min (cid:110) 1 , f ( (cid:104) φ ( s, a ) , ˆ θ h +1 ,t (cid:105) )
+ γ (cid:107) φ ( s, a ) (cid:107) Λ − 1 h
+1 ,t (cid:111) .
In our analysis, we verify that ¯ Q h +1 ,t is optimistic in the sense that it over-estimates Q (cid:63) for every ( s, a ) .
Then, the regression targets for the least squares problem at time point h are r h,τ + max a (cid:48)
∈A ¯ Q h +1 ,t ( s h
+1 ,τ , a (cid:48) ) , which is a natural stochastic approximation to the Bellman backup of ¯ Q h +1 ,t .
Applying this update backward from time point H to 1 , we obtain the Q -function estimates that we use to deﬁne the policy for the next episode.
The main conceptual difference between Algorithm 1 and the algorithm of Jin et al.
(2019) is that we allow non-linear function approximation with GLMs, while they consider only linear models.
On a more technical level, we use constrained least squares for our dynamic programming backup which we ﬁnd easier to analyze, while they use the ridge regularized version.
On the computational side, the algorithm is straightforward to implement, and, depending on the link function f , it can be easily shown to run in polynomial time.
For example, if f is the identity map, then (1) is equivalent to standard least square ridge regression, which can be solved in closed form.
Moreover, we can use the Sherman-Morrison formula to amortize matrix inversions, and, by doing so, we obtain a running time of O (cid:0) d 2 |A| HT 2 (cid:1) .
The dominant cost in this calculation is evaluating the optimism bonus when computing the regression targets.
In practice, using an epoch schedule or incremental optimization algorithms for updating ¯ Q would yield an even faster algorithm.
Of course, with modern machine learning libraries, it is also straightforward to implement the algorithm with a non-trivial link function f , even though (1) may be non-convex.
4. M AIN RESULT
Our main result is a regret bound for LSVI-UCB when the link function satisﬁes Assumption 1 and the function class satisﬁes Assumption 2.
Theorem 1.
For any episodic MDP, with Assumption 1 and Assumption 2, and for any T , the cumulative regret of Algorithm 1 is
[Equation]
with probability 1 − 1 / ( TH ) .
The result states that LSVI-UCB enjoys √ T -regret for any episodic MDP problem and any GLM, provided that the regularity conditions are satisﬁed and that optimistic closure holds.
As we have mentioned, these assumptions are relatively mild, encompassing the tabular setting and prior work on linear function approximation.
Importantly, no explicit dynamics assumptions are required.
Thus, Theorem 1 is one of the most general results we are aware of for provably efﬁcient exploration with function approximation.
Nevertheless, to develop further intuition for our bound, it is worth comparing to prior results.
First, in the linear MDP setting of Jin et al.
(2019), we use the identity link function so that K = κ = 1 and
M = 1 , and we also are guaranteed to satisfy Assumption 2.
In this case, our bound differs from that of Jin et al.
(2019) only in the dependence on H , which arises due to a difference in normalization.
Our bound is essentially equivalent to theirs and can therefore be seen as a strict generalization.
To capture the tabular setting, we use the standard basis featurization as in Fact 2 and the identity link function, which gives d = |S||A| , K = κ = 1 , and M = 1 .
Thus, we obtain the following corollary:
Corollary 2.
For MDPs with ﬁnite state and action spaces, using feature map φ ( s, a ) (cid:44) e s,a ∈
R |S|×|A| , for any T , the cumulative regret of Algorithm 1 is (cid:101) O (cid:16) H (cid:112) |S| 3 |A| 3 T (cid:17) , with probability 1 − 1 / ( TH ) .
Note that this bound is polynomially worse than the near-optimal (cid:101)
O ( H √ SAT + H 2 S
A log( T )) bound of Azar et al. (2017).
However, a reﬁned analysis specialized to the tabular setting can be shown to obtain a better regret bound of (cid:101) O (cid:16) H (cid:112) |S| 2 |A| 2 T (cid:17) .
Of course, our algorithm and analysis address problems with inﬁnitely large state spaces and other settings that are signiﬁcantly more complex than tabular MDPs, which we believe is more important than recovering the
D ISCUSSION
This paper presents a provably efﬁcient reinforcement learning algorithm that approximates the Q (cid:63) function with a generalized linear model.
We prove that the algorithm obtains (cid:101)
O
( H √ d 3 T ) regret under mild regularity conditions and a new expressivity condition that we call optimistic closure .
These assumptions generalize both the tabular setting, which is classical, and the linear MDP setting studied in recent work.
Further they represent the ﬁrst statistically and computationally efﬁcient algorithms for reinforcement learning with generalized linear function approximation, without explicit dynamics assumptions.
We close with some open problems.
First, using the fact that Corollary 3 applies beyond GLMs, can we develop algorithms that can employ general function classes?
While such algorithms do exist for the contextual bandit setting (Foster et al., 2018), it seems quite difﬁcult to generalize this analysis to multi-step reinforcement learning.
More importantly, while optimistic closure is weaker than some prior assumptions (and incomparable to others), it is still quite strong, and stronger than what is required for the batch RL setting.
An important direction is to investigate weaker assumptions that enable provably efﬁcient reinforcement learning with function approximation.
We look forward to studying these questions in future work.
A P ROOF OF T HEOREM
We now provide the proof of Theorem 1, deferring some technical details to later sections in this appendix.
The proof has three main components: a regret decomposition for optimistic Q learning, a deviation analysis for least squares with GLMs to ensure optimism and a potential argument to obtain the ﬁnal regret bound.
Regret decomposition.
The ﬁrst step of the proof is a regret decomposition that applies generically to optimistic algorithms.
The lemma demonstrates concisely the value of optimism in reinforcement learning and is the primary technical motivation for our interest in optimistic algorithms.
We state the lemma more generally, which requires some additional notation.
Fix round t and let { ¯ Q h,t − 1 } h ≤ H denote the current estimated Q functions.
The precondition is that ¯ Q h,t − 1 is optimistic and has controlled overestimation.
Precisely, we assume that there exists a function cnf h,t − 1 : S ×
A → R + such that
[Equation]
We will verify that our estimates ¯ Q h, · satisfy these properties subsequently.
Before doing so, we state the regret decomposition lemma and an immediate corollary.
Lemma 1.
Fix episode t and let F t − 1 be the ﬁltration of { ( s h,τ , a h,τ , r h,τ ) } τ<t .
Assume that ¯
Q h,t − 1 satisﬁes (3) for some function cnf h,t − 1 .
Then, if π t = argmax a ∈A ¯ Q h,t − 1 ( · , a ) is deployed we have [Equation] 1 ( s
[Equation]
where E [ ζ t | F t − 1 ] = 0 and | ζ t | ≤ 2 H almost surely.
Corollary 3.
Assume that for all t , ¯ Q h,t − 1 satisﬁes (3) and that π t is the greedy policy with respect to ¯ Q h,t − 1 .
Then with probability at least 1 − δ , we have
[Equation]
Proof of Lemma 1.
Observe that
[Equation]
Throughout this calculation, s 1 ∼ µ .
The ﬁrst step here is by deﬁnition, the second uses the optimism property for ¯ Q 1 ,t − 1 .
The third uses that π t is the greedy policy with respect to ¯ Q 1 ,t − 1 while the fourth uses the upper bound on ¯ Q 1 ,t − 1 .
Finally we use the deﬁnition of the Bellman operator and the fact that π t is the greedy policy yet again.
Comparing this upper bound with the expected reward collected by π t we observe that r 1 cancels, and we get
[Equation]
E
[Equation]
At this point, notice that ¯ Q 2 ,t − 1 ( s 2 , π t ( s 2 )) is precisely what we alreacy upper bounded at time point h
=
and we are always considering the state-action distribution induced by π t .
Hence, repeating the argument for all h , we obtain
[Equation]
where ζ t (cid:44) (cid:80) Hh =1 ζ h,t and
[Equation]
, a h,t ) , which is easily seen to have the required properties.
The lemma states that if ¯ Q h,t − 1 is optimistic and we deploy the greedy policy π t , then the perepisode regret is controlled by the overestimation error of ¯ Q h,t − 1 , up to a stochastic term that enjoys favorable concentration properties.
Crucially, the errors are accumulated on the observed trajectory, or, stated another way, the cnf h,t − 1 is evaluated on the states and actions visited during the episode.
As these states and actions will be used to update ¯ Q , we can expect that the cnf function will decrease on these arguments.
This can yield one of two outcomes: either we will incur lower regret in the next episode, or we will explore the environment by visiting new states and actions.
In this sense, the lemma demonstrates how optimism navigates the exploration-exploitation tradeoff in the multi-step RL setting, analogously to the bandit setting.
Note that Lemma 1 does not assume any form for ¯ Q h,t − 1 and does not require Assumption 2.
In particular, they are not specialized to GLMs.
In our proof, we use the GLM representation and Assumption 2 to ensure that (3) holds and to bound the conﬁdence sum in Corollary 3.
We believe these technical results will be useful in designing RL algorithms for general function classes, which is a natural direction for future work.
Deviation analysis.
The next step of the proof is to design the cnf function and ensure that (3) holds, with high probability.
This is the contents of the next lemma.
Lemma 2.
Under Assumption 1 and Assumption 2, with probability 1 − 1 / ( TH ) , we have that ∀ t, h, s, a :
[Equation]
where γ, Λ h,t are deﬁned in Algorithm 1.
A simple induction argument then veriﬁes that (3) holds, which we summarize in the next corollary.
Corollary 4.
Under Assumption 1 and Assumption 2, with probability 1 − 1 / ( TH ) , we have that (3) holds for all t, h with cnf h,t − 1 ( s, a )
=
min { 2 , 2 γ (cid:107) φ ( s, a ) (cid:107) Λ − 1 h,t − 1 } .
As the proof of Lemma 2 is rather long and technical, we defer the details to the appendix and instead explain the high-level argument here.
The proof requires an intricate deviation analysis to account for the dependency structure in the data sequence.
The intuition is that, thanks to Assumption 2 and the fact that ¯ Q h +1 ,t ∈ G up , we know that there exists a parameter ¯ θ h,t such that f ( (cid:104) φ ( s, a ) , ¯ θ h,t (cid:105) ) =
T h ( ¯ Q h +1 ,t )( s, a ) .
It is easy to verify that ¯ θ h,t is the Bayes optimal predictor for the square loss problem in (1), and so with a uniform convergence argument we can expect that ˆ θ h,t is close to ¯ θ h,t , which is our desired conclusion.
There are two subtleties with this argument.
First, we want to show that ¯ θ h,t and ˆ θ h,t are close in a data-dependent sense, to obtain the dependence on the Λ − 1 h,t -Mahalanobis norm in the bound.
This can be done using vector-valued self-normalized martingale inequalities (Pe˜na et al., 2008), as in prior work on linear stochastic bandits (Abbasi-Yadkori et al., 2012; Filippi et al., 2010; AbbasiYadkori et al., 2011).
However, the process we are considering is not a martingale, since ¯ Q h +1 ,t , which determines the regression targets y h,τ , depends on all data collected so far.
Hence y h,τ is not measurable with respect to the ﬁltration F τ , which prevents us from directly applying a self-normalized martingale concentration inequality.
To circumvent this issue, we use a uniform convergence argument and introduce a deterministic covering of G up .
Each element of the cover induces a different sequence of regression targets y h,τ , but as the covering is deterministic, we do obtain martingale structure.
Then, we show that the error term for the random ¯ Q h +1 ,t that we need to bound is close to a corresponding term for one of the covering elements, and we ﬁnish the proof with a uniform convergence argument over all covering elements.
The corollary is then obtained by a straightforward inductive argument.
Assuming ¯ Q h +1 ,t dominates
Q (cid:63) , it is easy to show that ¯ Q h,t also dominates Q (cid:63) , and the upper bound is immediate.
Combining Corollary 4 with Corollary 3, all that remains is to upper bound the conﬁdence sum.
Potential argument.
To bound the conﬁdence sum, we use a standard potential argument that appears in a number of works on stochastic linear bandits.
We summarize the conclusion with the following lemma, which follows directly from Lemma 11 of Abbasi-Yadkori et al.
Lemma 3.
For any h ≤ H we have that
[Equation]
Wrapping up.
Equipped with the above results, we are now prepared to prove Theorem
[Equation]
Proof of Theorem 1.
Assume that Corollary 4 holds for all 1 ≤ h ≤ H and 1 ≤ t ≤ T .
Applying Lemma 1 and the deﬁnition of cnf h,t − 1 implied by Corollary 4, the cumulative expected regret is at most
[Equation]
Here, the second step follows from the Cauchy-Schwarz inequality, and the last step is an application of Lemma 3.
The ﬁrst term forms a martingale, and we know that | ζ t | ≤ 2 H .
Therefore, by Azuma’s inequality, we have that with probability at least 1 − 1 /TH
[Equation]
Finally, using the deﬁnition of γ , the ﬁnal regret is upper bounded by
[Equation]
which proves the result.
B P ROOF OF L EMMA 2 AND C OROLLARY
To facilitate our analysis we deﬁne the following important intermediate quantity:
[Equation]
In words, ¯ θ h,t is the Bayes optimal predictor for the squared loss problem at time point h in the t th episode.
Since by inspection ¯ Q h +1 ,t ∈ G up , by Assumption 2 we know that ¯ θ h,t exists for all h and t .
Lemma 4.
For any θ, θ (cid:48) , x ∈ R d satisfying
[Equation]
≤
[Equation]
Proof.
By the mean-value theorem, there exists ˜ θ
= θ
+ λ ( θ (cid:48) − θ ) for some λ ∈ (0 , 1) such that f ( (cid:104) x, θ (cid:48) (cid:105) )
− f ( (cid:104) x, θ (cid:105) )
= (cid:68) ∇ θ
f
( (cid:104) x, ˜ θ (cid:105) ) , θ (cid:48) − θ (cid:69) .
On the other hand, by the chain rule and Assumption 1, ∇ θ f ( (cid:104) x, ˜ θ (cid:105) )
=
f (cid:48)
( (cid:104) x, ˜ θ (cid:105) )
· x .
Hence,
[Equation]
which are to be demonstrated.
Lemma 5.
For any 0 < ε ≤ 1 , there exists a ﬁnite subset V ε ⊂ G up with ln |V ε | ≤ 6 d 2 ln(
[Equation]
+
K + Γ) /ε ) , such that
[Equation]
Proof.
Recall that for every g ∈ G up , there exists θ ∈ B d , 0 ≤ γ ≤ Γ and (cid:107) A (cid:107) op ≤ 1 such
that g ( x ) =
min { 1 , f ( (cid:104) x, θ (cid:105) )
+ γ (cid:107) x (cid:107) A } .
Let Θ ε ⊆ B d , Γ ε ⊆
[0 , Γ] and M ε ⊆ { M ∈ S + d : (cid:107) M (cid:107) op ≤ 1 } be ﬁnite subsets such that for any θ, γ, A , there exist θ (cid:48)
∈
Θ ε , γ (cid:
) ∈
Γ ε , A (cid:48) ∈
M ε such that
[Equation]
where ε (cid:48) ∈ (0 , 1) will be speciﬁed later in the proof.
For the function g ∈ G up corresponding to the parameters θ, γ, A the function g (cid:48) corresponding to parameters θ (cid:48) , γ (cid:48) , A (cid:48) satisﬁes
[Equation]
In the last step we use ε (cid:48) ≤ 1 .
Therefore, if we deﬁne the class V ε (cid:44) { ( s, a ) (cid:55)→ min { 1 , f ( (cid:104) φ ( s, a ) , θ (cid:48) (cid:105) ) + γ (cid:48) (cid:107) φ ( s, a ) (cid:107) A
(cid:48) : θ (cid:
) ∈
Θ ε , γ ∈ Γ ε , A ∈ M ε } , we know that the covering property is satisﬁed with parameter (1 + K + Γ) √ ε (cid:48) .
Setting ε (cid:48)
=
ε 2 / (1 + K + Γ) 2 we have the desired covering property.
Finally, we upper bound ln |V ε | .
By deﬁnition, we have that ln |V ε | ≤ ln | Θ ε | + ln | Γ ε | + ln |M ε | .
Furthermore, standard covering number bounds reveals that ln | Θ ε | ≤
d ln(2 /ε (cid:48) ) , ln | Γ ε | ≤ ln(1 /ε (cid:48) ) and ln |M ε | ≤ d 2 ln(2 /ε (cid:48) ) .
Plugging in the deﬁnition of ε (cid:48) yields the result.
For the next lemma, let F t − 1 (cid:44) σ ( { ( s h,τ , a h,τ , r h,τ ) } τ<t ) be the ﬁltration induced by all observed trajectories up to but not including time t .
Observe that ¯ Q · ,t − 1 and our policy ˆ π h,t are F t − 1 measurable.
Lemma 6 (Restatement of Lemma 2) .
Fix any 1 ≤ t ≤ T and 1 ≤ h ≤ H .
Then as long as π t is
F t − 1 measurable, with probability 1 − 1 / ( TH ) 2 it holds that
[Equation]
∀
[Equation]
for γ ≥ CKκ [Equation] ln
[Equation]
K [Equation] TH ) and 0 < C < ∞ is a universal constant.
Note that this is precisely Lemma 2, as ¯ θ h,t is deﬁned as f ( (cid:104) φ ( s, a ) , ¯ θ h,t ) =
T h ( ¯ Q h +1 ,t )( s, a ) .
Proof.
The upper bound of 2 is obvious, since both terms are upper bounded by 1 in absolute value.
Therefore we focus on the second term in the minimum.
To simplify notation we omit the dependence on h in the subscripts and write x τ , y τ for x h,τ and y h,τ .
We also abbreviate ˆ θ (cid:44) ˆ θ h,t and ¯ θ (cid:44) ¯ θ h,t .
Since (cid:13)(cid:13) ¯ θ (cid:13)(cid:13) 2 ≤ 1 , the optimality of ˆ θ for (1) implies that
[Equation]
Decomposing the squares and re-organizing the terms, we have that
[Equation]
(5) where ξ τ (cid:44) y τ − f ( (cid:104) x τ , ¯ θ (cid:105) ) .
By the fundamental theorem of calculus
[Equation]
we have
[Equation]
Using this identity on both sides of (5)
[Equation]
we have that
[Equation]
Note also that, by Assumption 1, D τ satisﬁes κ 2 ≤ D 2 τ ≤ K 2 almost surely for all τ .
The difﬁculty in controlling (6) is that ¯ θ itself is a random variable that depends on { ( x τ , y τ ) }
τ ≤ t .
In particular, we want that
E [ ξ τ | D τ (cid:104) x τ , φ (cid:105) , F τ − 1 ] = 0 for any ﬁxed φ , but this is not immediate as ¯ θ depends on x τ .
To proceed, we eliminate this dependence with a uniform convergence argument.
Let ε ∈
(0 , 1) be a covering accuracy parameter to be determined later in this proof.
Let V ε be the pointwise covering for G up that is implied by Lemma 5.
Let g ε ∈ V ε be the approximation for ¯
Q h +1 ,t that satisﬁes (4).
By Assumption 2, there exists some θ (cid:93)
∈
B d such that
[Equation]
Now, deﬁne y (cid:93)τ and ξ (cid:93)τ as
[Equation]
The right-hand side of (6) can then be upper bounded as
[Equation]
(7) where | ∆ | ≤ Kt × max τ ≤ t | ξ (cid:93)τ − ξ τ | almost surely.
Upper bounding ∆ in (7).
Fix τ ≤ t .
By deﬁnition, we have that
[Equation]
where (8) holds by Lemma 4 and (9) follows from Lemma 5.
In particular, the bound on (cid:13)(cid:13) ¯ θ − θ (cid:93) (cid:13)(cid:13)can be veriﬁed by expanding the deﬁnitions and noting that g ε is pointwise close to ¯ Q h
+1 ,t .
Therefore, we have
[Equation]
Upper bounding (7).
Note that D τ is a function of x τ , ˆ θ , and ¯ θ .
For clarity, we deﬁne
[Equation]
for all | z | ≤ 1 and (cid:107) x τ (cid:107) 2 ≤ 1 , we have that for every θ, θ [Equation]
[Equation]
d
[Equation]
Hence, for any ( θ, θ (cid:48) ) and (˜ θ, ˜ θ (cid:48) ) pairs [Equation] we have for every τ that
[Equation]
[Table]
Here we are using that | ξ τ | ≤ 1 .
We are now in a position to invoke Lemma 8.
Consider a ﬁxed function g ε , which deﬁnes a ﬁxed θ (cid:93) .
We will bound (cid:12)(cid:12)(cid:12)(cid:80) τ ≤ t ξ (cid:93)τ (cid:104) x τ , D τ ( θ, θ (cid:48) )
( θ − θ (cid:48) ) (cid:105) (cid:12)(cid:12)(cid:12) uniformly over all pairs ( θ, θ (cid:48) ) .
With g ε , θ (cid:93) ﬁxed and since π t is F t − 1 measurable, we have that { x τ , ξ (cid:93)τ } τ ≤ t are random variables satisfying
E [ ξ (cid:93)τ | x 1: τ , ξ (cid:93) 1: τ − 1 ] = 0 .
For φ = ( θ, θ (cid:48) )
we deﬁne the function q ( x τ , φ ) =
(cid:104)
x, D τ ( φ )( θ − θ (cid:48) ) (cid:105) , which as we have just calculated satisﬁes | q ( x τ , φ )
− q ( x τ , φ (cid:48) )
| ≤ (2 M + K ) (cid:107) φ − φ (cid:48) (cid:107) 2 .
For δ (cid:48) ∈ (0 , 1 / 2) with probability 1 − δ (cid:48) we have ∀ φ =
( θ, θ (cid:48) )
∈
B 2 d : (cid:12)(cid:12) (cid:12)(cid:12)(cid:12)(cid:12)(cid:88) τ ≤ t ξ (cid:93)τ (cid:104) [Equation] [Equation] (cid:48)
[Equation]
(cid:105)
[Equation] + [Equation] (cid:48) ) (cid:111) , (11) where V ( φ ) (cid:44) (cid:80) τ ≤ t (cid:104) x
τ , D τ ( φ )( θ − θ (cid:48) ) (cid:105) 2 .
The last inequality holds because a + b ≤ 2 max { a, b } .
Next, take a union bound over all g ε ∈ V ε so (11) holds for any g ε and any subsequently induced choice of ξ (cid:93)τ with probability at least 1 − |V
ε | δ (cid:48) .
In particular, this union bound implies that (11) holds for the choice of g ε that approximates ¯ Q h +1 ,t .
Therefore, combining (6), (7), (10) with (11) for this choice of g ε , we have that with probability at least 1 − |V ε | δ (cid:48)
[Table]
Observe that the left hand side is precisely V (ˆ θ, ¯ θ ) .
Now, set ε = 1 / (2( K + 1) 2 T ) and δ (cid:48)
= 1 / ( |V ε | T 2 H 2 ) and use the bound on ln |V ε | from Lemma 5 to get
[Equation]
Therefore, we obtain
V
[Equation]
Subsequently,
[Equation]
where 0 < C V < ∞ is a universal constant.
Next, note that D 2 τ ≥ κ 2 , thanks to Assumption 1.
We then have
[Equation]
+
[Equation]
+ [Equation] ln((1 + K + Γ) TH ) , where Λ h,t = (cid:80) τ<t x τ , x (cid:62) τ .
Finally, for any ( s, a ) pair, invoking Lemma 4 and the Cauchy-Schwarz inequality we have
[Equation]
which is to be demonstrated
[Equation]
Corollary 5 (Restatement of Corollary 4) .
With probability 1 − 1 / ( TH ) , ¯ Q h,t ( s, a ) ≥ Q (cid:63)h ( s, a ) holds for all h, t, s, a .
Proof.
Fix 1 ≤ t ≤ T .
We use induction on h to prove this corollary.
For h =
H +1 , ¯ Q H +1 ,t ( · , · ) ≥
Q (cid:63) H +1 ( · , · ) clearly holds because ¯ Q H +1 ,t ≡ Q (cid:63) H +1 ≡ 0 .
Now assume that ¯ Q h +1 ,t ≥ Q (cid:63) h +1 , and let us prove that this is also true for time step h .
Since ¯ Q h +1 ,t ( s (cid:48) , a (cid:48) ) ≥ Q (cid:63)h +1 ( s (cid:48) , a (cid:48) ) for all s (cid:48) , a (cid:48) , we have that f ( (cid:104) φ ( s, a ) , ¯ θ h,t (cid:105) )
≥
f
( (cid:104) φ ( s, a ) , θ (cid:63)h (cid:105) ) for all ( s, a ) pairs.
Then, by the deﬁnition of ¯ Q h,t and Lemma 6, with probability 1 − 1 / ( TH )it holds uniformly for all ( s, a ) pairs that ¯ Q h,t ( s, a ) ≥ f
( (cid:104) φ ( s, a ) , ¯ θ h,t (cid:105) ) .
Hence, with the same probability, we have ¯ Q h,t ( s, a ) ≥ Q (cid:63)h ( s, a ) for all ( s, a ) pairs.
A union bound over all t ≤ T and h ≤ H completes the proof.
C T AIL INEQUALITIES
Lemma 7 (Azuma’s inequality) .
Suppose X 0 , X 1 , X 2 , · · · , X N form a martingale (i.e., E [ X k +1 | X 1 , · · · , X k ] =
X k ) and satisfy | X k − X k − 1 |
≤ c k almost surely.
Then for any (cid:15) > 0 ,
Pr (cid:2)(cid:12)(cid:12) X n − X 0 (cid:12)(cid:12) ≥ (cid:15) (cid:3) ≤ 2 exp (cid:40) − (cid:15)
[Equation]
Lemma 8.
Fix t, D ∈
N .
Let { ξ τ , u τ } τ ≤ t be random variables such that E [ ξ τ | u 1 , ξ 1 , · · · , u τ − 1 , ξ τ − 1 , u τ ] = 0 and | ξ τ | ≤ 1 almost surely.
Let q : ( u, φ ) (cid:55)→ R be an arbitrary deterministic function satisfying | q ( u, φ ) − q ( u, φ (cid:48) )
| ≤ C (cid:107) φ − φ (cid:48) (cid:107) 2 for all u, φ and φ (cid:48) , where φ, φ (cid:48) ∈
R D .
Then for any δ ∈ (0 , 1) and R > 0 ,
Pr
[Equation]
where B D ( R ) (cid:44) { x ∈ R D : (cid:107) x (cid:107) 2 ≤ R } and V q ( φ ) (cid:44) (cid:80) τ ≤ t q 2 ( u τ , φ ) .
Proof.
Let (cid:15) > 0 be a small precision parameter to be speciﬁed later.
Let H ⊆
B D ( R ) be a ﬁnite (cid:15) - covering of B D ( R ) such that sup x ∈ B D ( R ) min z ∈H (cid:107) x
− z (cid:107) 2 ≤ (cid:15) .
Using standard covering number arguments, such a covering exists with ln |H| ≤ D ln(2 R/(cid:15) ) .
For any φ ∈ B D ( R ) let φ (cid:48)
(cid:44) argmin z ∈H (cid:107) φ − z (cid:107) 2 .
By deﬁnition, (cid:107) φ − φ (cid:48) (cid:107) 2 ≤ (cid:15) .
This implies (cid:12)(cid:12)(cid:12)(cid:80) tτ =
1 ξ τ [ q ( u τ , φ ) − q ( u τ , φ (cid:48) )] (cid:12)(cid:12)(cid:12) ≤ Ct(cid:15) because | ξ τ | ≤ 1 almost surely.
Subsequently, for any ∆ > 0 ,
Pr
[Equation]
≤ (cid:88)
[Equation]
where the last inequality holds by the union bound.
For any ﬁxed φ (cid:48) ∈ H , h ( u τ , φ (cid:48) ) only depends on u τ , and therefore E [ ξ τ | q ( u τ , φ (cid:48) )]
= 0 for all τ .
Invoking Lemma 7 with X τ (cid:44) (cid:80) τ (cid:48) ≤ τ ξ τ (cid:48) q ( u τ (cid:48) , φ (cid:48) ) and c τ (cid:48)
=
| q ( u τ (cid:48) , φ (cid:48) )
| , we have
[Equation]
Equating the right-hand side of the above inequality with δ (cid:48) and combining with the union bound application, we have
[Equation]
Further equating δ (cid:48)
=
δ/ |H| and using the fact that ln |H| ≤ D ln(2 R/(cid:15) ) , we have
Pr
[Equation]
(cid:35) ≤ δ.
Finally, as | q ( u τ , φ (cid:48) ) − q ( u τ , φ ) | ≤ (cid:15) , we have V q ( φ (cid:48) ) ≤ 2 V q ( φ ) + 2 t(cid:15) 2 and so
Pr
[Equation]
≤ δ.
Setting (cid:15) = 1 /t in the above inequality completes the proof.