L EARNING S TOCHASTIC S HORTEST P ATH WITH L INEAR F UNCTION A PPROXIMATION
Anonymous authors
Paper under double-blind review
A BSTRACT
We study the stochastic shortest path (SSP) problem in reinforcement learning with linear function approximation, where the transition kernel is represented as a linear mixture of unknown models.
We call this class of SSP problems as linear mixture
SSP.
We propose a novel algorithm for learning the linear mixture SSP which can attain a (cid:101) O ( dB 1 . 5 (cid:63) (cid:112) K/c min ) regret.
Here K is the number of episodes, d is the dimension of the feature mapping in the mixture model, B (cid:63) bounds the expected cumulative cost of the optimal policy and c min > 0 is the lower bound of the cost function.
Our algorithm also applies to the case when c min = 0 , where a (cid:101) O
( K 2 / 3 ) regret is guaranteed.
To the best of our knowledge, this is the ﬁrst algorithm with a sublinear regret guarantee for learning linear mixture SSP.
In complement to the regret upper bounds, we also prove a lower bound of Ω( dB (cid:63) √ K ) , which nearly matches our upper bound.
I NTRODUCTION
The Stochastic Shortest Path (SSP) model refers to a type of reinforcement learning (RL) problems where an agent repeatedly interacts with a stochastic environment and aims to reach some speciﬁc goal state while minimizing the cumulative cost.
Compared with other popular RL settings such as episodic and inﬁnite-horizon Markov Decision Processes (MDPs), the horizon length in SSP is random, varies across different policies, and can potentially be inﬁnite because the interaction only stops when arriving at the goal state.
Therefore, the SSP model includes both episodic and inﬁnite-horizon MDPs as special cases, and is comparably more general and of broader applicability.
In particular, many goal-oriented real-world problems ﬁt better into the SSP model, such as navigation and GO game (Andrychowicz et al., 2017; Nasiriany et al., 2019).
In recent years, there emerges a line of works on developing efﬁcient algorithms and the corresponding analyses for learning SSP.
Most of them consider the episodic setting, where the interaction between the agent and the environment proceeds in K episodes (Cohen et al., 2020; Tarbouriech et al., 2020a).
For tabular SSP models where the sizes of the action and state space are ﬁnite, Cohen et al.
(2021) developed a ﬁnite-horizon reduction algorithm that achieves the minimax regret (cid:101)
O ( B (cid:63) √ SAK ) , where B (cid:63) is the largest expected cost of the optimal policy starting from any state, S is the number of states and A is the number of actions.
In a similar setting, Tarbouriech et al.
(2021b) proposed the ﬁrst algorithm that is minimax optimal, parameter-free and horizon-free at the same time.
However, the algorithms mentioned above only apply to tabular SSP problems where the state and action space are small.
In order to deal with SSP problems with large state and action spaces, function approximation techniques (Yang & Wang, 2019; Jin et al., 2020;
Jia et al., 2020;
Zhou et al., 2021b; Wang et al., 2020b;a) are needed.
Following the recent line of work on model-based reinforcement learning with linear function approximation (Modi et al., 2020;
Jia et al., 2020; Ayoub et al., 2020;
Zhou et al., 2021b), we consider a linear mixture SSP model, which extends the tabular SSP.
More speciﬁcally, we assume that the transition probability is parametrized by P ( s (cid:48) | s, a ) = (cid:104) φ ( s (cid:48) | s, a ) , θ ∗ (cid:105) for all triplet ( s, a, s (cid:48)
) ∈
S ×
A × S , where S is the state space and A is the action space.
Here we assume that φ ∈ R d is a known ternary feature mapping, and θ ∗ ∈ R
d is an unknown model parameter vector that needs to be learned.
Such a setting has been previously studied for episodic MDPs (Modi et al., 2020;
Jia et al., 2020; Ayoub et al., 2020;
Cai et al., 2020) and inﬁnite-horizon discounted MDPs (Zhou et al., 2021b).
Nevertheless, algorithms developed in these works do not apply to SSP since the horizon length is random as mentioned above.
To tackle the challenge of varying horizon length, we propose a model-based optimistic algorithm with linear function approximation, dubbed LEVIS , for learning the linear mixture SSP.
At the core of our algorithm are a conﬁdence set of the model parameters and a specially designed Extended
Value Iteration ( EVI ) subroutine for computing the optimistic estimate of the value function, which together guarantee that the algorithm will reach the goal state in every episode.
Compared with the
EVI subroutine developed for inﬁnite-horizon discounted MDPs (Zhou et al., 2021b), we introduce a shrinking factor q ≈ 1 /t in our EVI with t being the cumulative number of time steps, which guarantees the convergence of EVI .
To compensate for the bias introduced by this shrinking factor, our algorithm performs lazy policy update, which is triggered by the doubling of the time interval between two policy updates or the doubling of the determinant of the covariance matrix.
With all these algorithmic designs, our algorithm is guaranteed to achieve a (cid:101) O ( dB 1 . 5 (cid:63) (cid:112) K/c min ) regret when c min > 0 .
To the best of our knowledge, this is the ﬁrst algorithm that enjoys a sublinear regret for linear mixture SSP.
It is worth noting that a recent work by Vial et al.
(2021) studied a different linear SSP model that is similar to linear MDPs (Yang & Wang, 2019; Jin et al., 2020), where both the underlying transition probability and cost function are linear in a known d -dimensional feature mapping ψ ∈ R d , i.e., P ( s (cid:48) | s, a ) = (cid:104) ψ ( s, a ) , µ ( s ) (cid:105) and
c ( s, a ) = (cid:104) ψ
( s, a ) , θ (cid:105) , and µ ( · ) and θ are unknown.
They proposed an algorithm with linear function approximation, which achieves (cid:101) O
( (cid:112) B 3 (cid:63) d 3 K/c min ) regret.
The linear SSP model is different from our model, and we refer the interested readers to Ayoub et al.
; Zhou et al.
(2021b)
for a detailed comparison between these two assumptions.
Besides the model difference, Vial et al.
(2021) further assumed the feature mapping to be orthonormal in order to obtain the (cid:101) O ( √ K ) regret.
We do not need such restrictive assumptions on the feature mapping, thus our algorithm provably works for more general cases.
Our contributions are summarized as follows:
• We propose to study a linear mixture SSP model, and devise a novel algorithm, dubbed L ower conﬁdence E xtended V alue I teration for S SP ( LEVIS ), for learning SSP with linear function approximation.
• We prove that LEVIS achieves a regret of order (cid:101)
O
( B 1 . 5 (cid:63) d (cid:112) K/c min ) when c min > 0 and the agent has an order-accurate estimate B ≥ B (cid:63) 1 .
For the general case where c min = 0 , our algorithm can achieve (cid:101) O
( K 2 / 3 ) regret guarantee by using a cost perturbation trick (Tarbouriech et al., 2021b).
• We prove that for linear mixture SSP, the regret of any learning algorithms is at least Ω( dB (cid:63) √ K ) .
This suggests that when c min > 0 , our algorithm is optimal with regard to the dimension of the feature mapping d and number of episodes K .
Notation We use lower case letters to denote scalars, and use lower and upper case bold face letters to denote vectors and matrices respectively.
For any positive integer n , we denote by [ n ] the set { 1 , . . . , n } .
For a vector x ∈ R d , we denote by (cid:107) x (cid:107) 1 the Manhattan norm and denote by (cid:107) x (cid:107)
the
Euclidean norm.
For a vector x ∈ R d and matrix Σ ∈ R d × d , we deﬁne (cid:107) x (cid:107) Σ = √ x (cid:62) Σx .
For two sequences { a n } and { b n } , we write a n = O ( b n ) if there exists an absolute constant C such that a n ≤ Cb n .
We use (cid:101) O ( · ) to hide the logarithmic factors.
R ELATED W ORK
Online learning in SSP SSP problems can be dated back to (Bertsekas & Tsitsiklis, 1991; Bertsekas & Yu, 2013; Bertsekas, 2012), but it is until recently that the regret minimization in online learning of SSP has been studied.
In the tabular case, Tarbouriech et al.
(2020a) proposed the ﬁrst algorithm achieving a (cid:101)
O ( D 3 / 2 S (cid:112) AK/c min ) regret where D is the diameter of SSP 2 .
The regret was further improved to (cid:101) O
( B (cid:63) S √ AK ) by Rosenberg et al.
; Cohen et al.
(2020), with an extra √ S factor compared with the Ω( B (cid:63) √ SAK )
lower bound (Rosenberg et al., 2020).
More recently, the (cid:101) O
( B (cid:63) √ SAK ) minimax optimal regret were obtained by Cohen et al.
(2021) and Tarbouriech et al.
(2020b) independently using different approaches.
Speciﬁcally, Cohen et al.
(2021) reduced SSP to a ﬁnite-horizon MDP with a large terminal cost assuming B (cid:63) is known; while Tarbouriech et al.
(2021b) avoid such requirement by adaptively estimating B (cid:63) with a doubling trick, together with a value iteration sub-routine ensuring the optimistic estimate of the value function.
Our proposed method shares a similar spirit with the latter approach, but for learning SSP with linear function approximation.
The above algorithms are all model-based.
Very recently, Chen et al.
(2021a) developed the ﬁrst modelfree algorithm for SSP which achieves the minimax optimal regret when the minimum cost among all state-action pairs c min is strictly positive.
Their method is motivated by the UCB-ADVANTAGE algorithm (Zhang et al., 2020).
For other settings of SSP, (Rosenberg & Mansour, 2020;
Chen &
Luo, 2021; Chen et al., 2021b) studied the case of adversarial costs.
Also, the pioneering work by (Bertsekas & Tsitsiklis, 1991) studied the pure planning problem in SSP where the agent has full knowledge of all the model parameters, and is followed by a series of works (Bonet, 2007; Kolobov et al., 2011; Bertsekas & Yu, 2013; Guillot & Stauffer, 2020).
On the other hand, Tarbouriech et al.
(2021a) studied the sample complexity of SSP assuming the access to a generative model.
Jafarnia-Jahromi et al.
(2021) proposed the ﬁrst posterior sampling algorithm for SSP.
Multi-goal
SSP have also been studied by Lim & Auer (2012); Tarbouriech et al.
(2020b).
Linear function approximation Linear MDP is one of the most widely studied models for RL with linear function approximation, which assumes both the transition probability and reward functions are linear functions of a known feature mapping (Yang & Wang, 2019; Jin et al., 2020).
Representative work in this direction include Du et al.
; Zanette et al.
; Wang et al.
(2020a)
; He et al.
(2021), to mention a few.
Another popular model for RL with linear function approximation is the so-called linear mixture
MDP/linear kernel MDP (Yang & Wang, 2020; Modi et al., 2020;
Jia et al., 2020; Ayoub et al., 2020;
Cai et al., 2020; Zhou et al., 2021b;a).
For the ﬁnite-horizon setting, Jia et al.
(2020) proposed a
UCRL-VTR algorithm that achieves a (cid:101) O
( d √ H 3 T ) regret bound.
Zhou et al.
(2021a) further improve the result by proposing a UCRL-VTR+ algorithm that attains the nearly minimax optimal regret (cid:101) O ( dH √ T ) based on a novel Bernstein-type concentration inequality.
For the discounted inﬁnite horizon setting, Zhou et al.
(2021b) proposed a UCLK algorithm with a (cid:101) O ( d √ T/ (1 − γ ) 2 ) regret, and also give a (cid:101)
O ( d √ T/ (1 − γ ) 1 . 5 ) lower bound.
The lower bound is later matched up to logarithmic factors by the UCLK+ algorithm
(Zhou et al., 2021a).
The SSP model studied in this paper can be seen as an extension of linear mixture MDPs.
P RELIMINARIES
Stochastic Shortest Path
An SSP instance is an MDP M := {S , A , P , c, s init , g } , where S and A are the ﬁnite state space and action space respectively.
Here s init denotes the initial state and g ∈ S is the goal state.
We denote the cost function by c : S ×
A → [0 , 1] , where c ( s, a ) is the immediate cost of taking action a at state s .
The goal state g incurs zero cost, i.e., c ( g, a ) =
for all a ∈ A .
For any ( s (cid:48) , s, a ) ∈ S ×
A × S , P ( s (cid:48) | s, a ) is the probability to transition to s (cid:48) given the current state s and action a being taken.
The goal state g is an absorbing state, i.e., P ( g | g, a )
= 1 for all action a ∈ A .
Linear mixture SSP
In this work, we assume the transition probability function P to be a linear mixture of some basis kernels (Modi et al., 2020;
Ayoub et al., 2020;
Zhou et al., 2021a).
Assumption 3.1.
Suppose the feature mapping φ : S
×
A × S → R d is known and pregiven.
There exists an unknown vector θ ∗ ∈ R d with (cid:107) θ ∗ (cid:107) 2 ≤ √ d such that P ( s (cid:48) | s, a ) = (cid:104) φ ( s (cid:48) | s, a ) , θ ∗ (cid:105) for any state-action-state triplet ( s, a, s (cid:48) )
∈ S ×
A × S .
Moreover, for any bounded function V : S → [0 , B ] , it holds that (cid:107) φ V ( s, a ) (cid:107) 2 ≤ B √ d for all ( s, a ) ∈ S ×A , where φ V ( s, a ) :=
(cid:80) s (cid:48) ∈S φ ( s (cid:48) | s, a ) V ( s (cid:48) ) .
For simplicity, for any function V : S → R , we denote P V ( s, a )
= (cid:80) s (cid:48)
P ( s (cid:48) | s, a ) V ( s (cid:48) ) for all ( s, a ) ∈ S × A .
Therefore, under Assumption 3.1, we have
[Equation]
Proper policies A stationary and deterministic policy is a mapping π : S
→ A such that the action π ( s ) is taken given the current state s .
We denote by T π ( s ) the expected time that it takes by following π to reach the goal state g starting from s .
We say a policy π is proper if T π ( s ) < ∞ for any s ∈ S (otherwise it is improper).
We denote by Π proper the set of all stationary, deterministic and proper policies.
We assume that Π proper is non-empty, which is the common assumption in previous works on online learning of SSP (Rosenberg et al., 2020; Rosenberg & Mansour, 2020;
Cohen et al., 2021; Tarbouriech et al., 2021b; Jafarnia-Jahromi et al., 2021;
Chen et al., 2021a).
Assumption 3.2.
The set of all stationary, deterministic and proper policies is non-empty, i.e., Π proper (cid:54) = ∅ .
Remark 3.3.
The above assumption is weaker than Assumption 1 in Vial et al.
(2021) which requires that all stationary policies are proper.
For any policy π , we deﬁne the cost-to-go function (a.k.a., value function) as
[Equation]
where s t +1 ∼ P (cid:0) · | s t , π ( s t ) (cid:1) .
V π ( s ) can possibly be inﬁnite if π is improper.
The action-value function of policy π is deﬁned as
[Equation]
where s 2 ∼ P ( ·| s 1 , a 1 ) and s t
+1 ∼ P ( ·| s t , π ( s t )) for all t ≥ 2 .
Since c ( · , · ) ∈ [0 , 1] , for any proper policy π ∈ Π proper , V π and Q π are both bounded functions.
Bellman optimality For any function V : S → R , we deﬁne the optimal Bellman operator L as
[Equation]
Intuitively speaking, we want to learn the optimal policy π (cid:63) such that V (cid:63) ( · ) :
= V π (cid:63) ( · ) is the unique solution to the Bellman optimality equation V =
L V and π (cid:63) minimizes the value function V π ( s ) component-wise over all policies.
It is known that, in order for such π (cid:63) to exist, one sufﬁcient condition is Assumption 3.2 together with an extra condition that any improper policy π has at least one inﬁnite-value state, i.e., for any π / ∈ Π proper , there exists some s ∈ S s.t.
V π ( s ) =
+ ∞ (Bertsekas & Tsitsiklis, 1991; Bertsekas & Yu, 2013; Tarbouriech et al., 2021b).
Note that this additional condition is satisﬁed in the case of strictly positive cost , where for any state s (cid:54) =
g and a ∈ A , it holds that c ( s, a ) ≥ c min .
To deal with the case of general cost function, one can adopt the cost perturbation trick (Tarbouriech et al., 2021b) and consider a modiﬁed problem with cost function
c ρ ( s, a ) :
= max { c ( s, a ) , ρ } for some ρ > 0 .
This will introduce an additional cost of order O ( ρT ) to the regret of the original problem, where T is the total number of steps.
Therefore, the second condition can be avoided, and we can assume the existence of π (cid:63) .
Throughout the paper, we denote by B (cid:63) the upper bound of the optimal value function V (cid:63) , i.e.,
B (cid:63) := max s ∈S V (cid:63) ( s ) .
Also, we deﬁne T (cid:63) :
=
max s ∈S T π (cid:63) ( s ) , which is ﬁnite under Assumption 3.2.
Since the cost is bounded by 1, we have B (cid:63) ≤ T (cid:63) <
+ ∞ .
Without loss of generality, we assume that
B (cid:63) ≥ 1 .
Furthermore, we denote the corresponding optimal action-value function by Q (cid:63) :
= Q
[Equation]
which satisﬁes the following Bellman equation for all ( s, a ) ∈ S
× A :
[Equation]
Learning objective Under Assumption 3.1, we assume c to be known for the ease of presentation.
We study the episodic setting where each episode starts from a ﬁxed initial state s init and ends only if the agent reaches the goal state g .
Given the total number of episodes, K , the objective of the agent is to minimize the regret over K episodes deﬁned as
[Equation]
where I k is the length of the k -th episode and c k,i =
c ( s k,i , a k,i ) is the cost triggered at the i -th step during the k -th episode.
Note that R K might be inﬁnite if some episode never ends.
A LGORITHMS
In this section, we propose a model-based algorithm named LEVIS , as displayed in Algorithm 1.
LEVIS is inspired by the UCLK -type of algorithms originally designed for discounted linear mixture
MDPs (Zhou et al., 2021a;b).
Our algorithm takes a multi-epoch form, where each episode is divided into epochs of different lengths (Jaksch et al., 2010; Lattimore & Hutter, 2012).
Within each epoch, the agent executes the greedy policy induced by some optimistic estimator of the optimal Q-function.
The switch between any two epochs is triggered by a doubling criterion, and then the estimated
Q-function is updated through an Extend Value Iteration ( EVI ) sub-routine (Algorithm 2).
We now give a detailed description of Algorithm 1.
1: Input: regularization parameter λ , conﬁdence radius { β t } , cost perturbation ρ ∈ [0 , 1] , an estimate B ≥ B (cid:63) 2: Initialize: set t ← 1 , j ← 0 , t 0 = 0 , Σ
← λ I , b 0 ← 0 , Q 0 ( s, · ) , V 0 ( s ) ← 1 ∀ s (cid:54) =
g andotherwise 3: for k = 1 , . . .
, K do 4:
Set s t = s init 5: while s t (cid:54) =
g do 6:
Take action a t = argmin a ∈A Q j ( s t , a ) , receive cost c t
= c ( s t , a t ) and next state s t +1 ∼ P ( ·| s t , a t ) 7:
Set Σ t ← Σ t − 1 + φ V j ( s t , a t ) φ V j ( s t , a t ) (cid:62) 8:
Set b t ← b t − 1 + φ V j ( s t , a t ) V j ( s t +1 ) 9: if det( Σ t ) ≥ 2 det( Σ t j ) or t ≥ 2 t j then 10:
Set j ← j +11:
Set t j ← t , (cid:15)
j ← 1 t j 12: (cid:98) θ j ← Σ − 1 t b t 13:
Set conﬁdence set C j ← (cid:110) θ : (cid:107) Σ 1 / 2 t j ( θ − (cid:98) θ j ) (cid:107) 2 ≤ β t j (cid:111) 14:
Set Q j ( · , · ) ← EVI ( C j , (cid:15) j , 1 t j , ρ ) 15:
Set V j ( · ) ← min a ∈A Q j ( · , a ) 16: end if 17:
Set t ← t +18: end while 19: end for
( i +1)
← c ρ ( · , · ) + (1 − q ) · min θ ∈C∩B (cid:104) θ , φ V ( i ) ( · , · ) i +
← min a ∈A Q i +1) a )
In Algorithm 1, we maintain two global indices.
Index t represents the total number of steps, and index j tracks the number of calls to the EVI sub-routine, where the output of EVI is an updated optimistic estimator of the optimal action-value function.
Each episode starts from a ﬁxed initial state s init (Line 4), ends when the goal state g is reached (Line 5) and is decomposed into epochs indexed by the global index j .
Within epoch j , the agent repeatedly executes the policy induced by the current estimation Q j of the action-value function (Line 6) and updates Σ t and b t (Lines 7 and 8).
The current epoch ends when the either criterion in Line 9 is triggered, and the EVI subroutine performs an optimistic planning to update the action-value function estimator (Lines 10 to 15).
Update criteria As mentioned before, Algorithm 1 runs in epochs indexed by j , and one epoch ends when either of the two update criteria is triggered (Line 9).
The ﬁrst updating criterion is satisﬁed once the determinant of Σ t is doubled compared to the determinant at the end of the previous epoch.
This is called lazy policy update that has been used in the linear bandits and RL literature (Abbasi-Yadkori et al., 2011; Zhou et al., 2021b), which reﬂects the diminishing return of learning the underlying transition.
One intuition behind the determinant doubling criterion is that the determinant can be viewed as a surrogate measure of the exploration in the feature space.
Thus, one only updates the policy when there is enough exploration being made since last update.
Moreover, this update criterion reduces the computational cost as the total number of epochs would be bounded by O (log T ) .
Here T denotes the total number of steps through all K episodes.
The doubling visitation criterion used in tabular SSP (Jafarnia-Jahromi et al., 2021; Tarbouriech et al., 2021b) can be viewed as a special case of this doubling determinant-based criterion.
However, the above criterion alone cannot guarantee ﬁnite length for each epoch as we do not have that (cid:107) φ V ( · , · ) (cid:107) is bounded from below, which holds for tabular SSP naturally since at most |S||A| max s ∈S ,a ∈A n ( s, a ) steps sufﬁce to double n ( s, a ) for at least a pair of s, a by the pigeonhole principle.
To address this problem, we show that we only need to add an extra triggering criterion:
t ≥ 2 t j .
It turns out that despite of being extremely simple this criterion endows the algorithm with several nice properties.
First, together with the EVI error parameter (cid:15) j = 1 /t j , we can bound the cumulative error from value iterations in epoch j by a constant, i.e., (2 t j − t j ) · (cid:15) j = 1 .
Second, it will not increase the total number of epochs since the time step doubling can happen at most O (log T ) times, which is consistent with the ﬁrst criterion.
These two properties together allow us to bound the total error from value iteration by O (log T ) .
Finally, this criterion is fairly easy to implement and has negligible time and space complexity.
Optimistic planning The optimism of Algorithm 1 is realized by the construction of the conﬁdence set C j (Line 11), which is fed into the EVI subroutine.
We now describe the construction of the
Q-function estimator in the EVI sub-routine (Algorithm 2).
EVI requires the access to a conﬁdence ellipsoid C j that contains the true model parameter θ ∗ with high probability (Line 13).
Here we construct the conﬁdence set C j centered at the minimizer of the ridge regression problem with a conﬁdence radius parameter β t (Line 13).
Since not every θ ∈ C
j deﬁnes a valid transition probability function, we further take the intersection between C j and a constraint set B deﬁned as follows
B := { θ : ∀ ( s, a ) , (cid:104) φ ( ·| s, a ) , θ (cid:105) is a probability distribution and (cid:104) φ ( s (cid:48) | g, a ) , θ (cid:105)
= 1 { s (cid:48) =
g }} .
Then C j ∩ B is still a conﬁdence set containing the true model parameter θ ∗ with high probability as θ ∗ ∈ B .
Algorithm 2 requires two additional inputs: optimality gap (cid:15) j and discount factor q .
The use of (cid:15) j is standard, but this discount factor is the key to ensuring convergence of EVI .
Speciﬁcally, (4.1) in Algorithm 2 repeatedly conducts one-step value iteration by applying the best possible Bellman operator to the set C j ∩ B .
This is motivated by the Bellman optimality equation in (3.2), and uses min θ ∈C∩B (cid:104) θ , φ V ( i ) (cid:105) as an optimistic estimate for P V ∗ .
However, using this estimate alone cannot guarantee the convergence of EVI because (cid:104)· , φ V ( i ) (cid:105) is not a contractive map, which holds for free in the discounted setting (Jaksch et al., 2010;
Zhou et al., 2021b), but not in SSP.
More speciﬁcally, in the EVI algorithm for the discounted setting (e.g., Algorithm 2 in (Zhou et al., 2021b)), there is an intrinsic discount factor 0 < γ < 1 , which ensures that the Bellman operator is a contraction.
As a result, the value iteration converges in a ﬁnite number of iterations.
However, the Bellman equation of SSP does not have a discount factor.
To address this issue, in (4.1), we introduce an extra 1 − q discount factor to ensure the contraction property.
Although this causes an additional bias to the estimated transition probability function, we can alleviate it by choosing q properly.
In particular, for each epoch
j we set q = 1 /t j
(Line 14), and as will be shown, this bias will only introduce an additive term of order
O (log T ) in the ﬁnal regret bound.
Besides the convergence guarantee, the 1 − q factor also brings an additional beneﬁt that it biases the estimated transition kernel towards the goal state g , further encouraging optimism.
Similar design can also be found in the VISGO value iteration algorithm used by Tarbouriech et al.
(2021b).
The intuition behind such a design is to ensure the existence of proper policies under the estimated transition probability function.
As a result, the output of the value iteration, which solves V = (cid:101)
L V approximately for the Bellman operator (cid:101)
L induced by the estimated transition, can induce a greedy policy that is proper under the estimated transition.
Regarding the implementation of LEVIS , note that the main computational overhead is from EVI , where within each inner iteration we need to solve an optimization problem.
Fortunately, the loss function is strongly convex, thus it can be efﬁciently solved by many convex optimization algorithms.
M AIN R ESULTS
In this section, we present the main theoretical results for Algorithm 1.
We provide regret upper bounds for both positive cost functions and general cost functions, followed by a lower bound.
5. U PPER B OUNDS : P OSITIVE C OST F UNCTIONS
We ﬁrst consider a special case where the cost is strictly positive (except for the goal state g ).
Assumption 5.1.
There exists an unknown constant c min ∈ (0 , 1) such that c ( s, a ) ≥ c min for all s ∈ S \ { g } and a ∈ A .
Let T be the total number of steps in Algorithm 1, then the above assumption allows us to lower bound the total cumulative cost after the K episodes by c min · T .
Note that this provides a relation between the deterministic K and the random quantity T .
To simplify the expression, we assume the agent has access to B , an order-accurate estimate of B (cid:63) satisfying B (cid:63) ≤ B ≤ κB (cid:63) for some unknown constant κ ≥ 1 .
Similar assumptions have also been imposed in previous works (Tarbouriech et al., 2021b;
Vial et al., 2021).
Theorem 5.2.
Under Assumptions 3.1, 3.2 and 5.1, for any δ > 0 , let ρ = 0 and β t =
B (cid:112) d log (4( t 2 + t 3 B 2 /λ ) /δ )
+ √ λd for all t ≥ 1 , where B ≥ B (cid:63) and λ ≥ 1 .
Then with probability at least 1 − δ , the regret of Algorithm 1 satisﬁes
[Equation]
If B = O ( B (cid:63) ) , Algorithm 1 attains an (cid:101) O
( B (cid:63) 1 .
5 d (cid:112) K/c min ) regret.
The dominating term in (5.1) has an dependency on 1 /c min .
For the tabular SSP, Cohen et al.
(2021); Jafarnia-Jahromi et al.
(2021); Tarbouriech et al.
(2021b) avoid such dependency by using a Bernstein-type conﬁdence set.
However, it remains an open question whether a similar result can be achieved under the linear function approximation setting.
Remark 5.3.
If we set the parameter δ in Theorem 5.2 as δ = 1 /K and deﬁne the high probability event Ω as Theorem 5.2 holds.
Then, for the expected regret, we have
[Equation]
which implies an (cid:101)
O ( B (cid:63) 1 .
5 d (cid:112) K/c min ) expected regret.
5. U PPER B OUND : G ENERAL C OST F UNCTIONS
When Assumption 5.1 does not hold, an (cid:101) O
( K 2 / 3 ) regret can be achieved by running Algorithmwith ρ = K − 1 / 3 .
Theorem 5.4.
Under Assumptions 3.1 and 3.2, for any δ > 0 , let ρ =
K − 1 / 3 and β t =
B (cid:112) d log (4( t 2 + t 3 B 2 /λ ) /δ )
+ √ λd for all t ≥ 1 , where B ≥ B (cid:63) and λ ≥ 1 .
Then with probability at least 1 − δ , the regret of Algorithm 1 satisﬁes
[Equation]
B 2 d 2 K 1 / 3 · χ (cid:17) , where (cid:101) B = B + T (cid:63) /K 1 / 3 and χ = log 2 (cid:0) ( B + T (cid:63) )
Kd/δ (cid:1) .
In Theorem 5.4, the regret depends on (cid:101) B instead of B .
Note that (cid:101) B is approximately equal to B (cid:63) when K = Ω( T 3 (cid:63) ) and B
= O
( B (cid:63) ) .
Here T (cid:63) is deﬁned in Section 3 as the maximum expected time it takes for the optimal policy to reach the goal state starting from any state.
The cost perturbation ρ is a common trick to deal with the case of general cost functions in the
SSP literature (Tarbouriech et al., 2020a; Cohen et al., 2020; Tarbouriech et al., 2021b).
Similar to Tarbouriech et al. (2020a), the term c − 1min is multiplicative with K in our regret bound given by
Theorem 5.2.
As a result, the perturbation can only give an (cid:101) O
( K 2 / 3 ) regret in the case of general cost functions.
Similarly, the regret bound of learning linear SSP (Vial et al., 2021) also has a multiplicative c − 1min .
Some later work on tabular SSP (Cohen et al., 2020; Tarbouriech et al., 2021b) has shown that it is possible to make the term c − 1min additive instead of multiplication, which improves the regret to (cid:101)
O ( K 1 / 2 ) for general cost functions.
How to get an additive c − 1min term in linear function approximation setting is an interesting future direction.
For the choice of the other parameters in Algorithm 1, by Theorems 5.2 and 5.4, we can set λ
=in both the positive and general cost cases.
For the upper bound B ≥ B (cid:63) , note that assuming a known B is common in existing SSP literature (Cohen et al., 2021; Vial et al., 2021).
Although it is possible to deal with an unknown B in the tabular SSP with a doubling trick (Rosenberg et al., 2020;
Tarbouriech et al., 2021b), it remains an open question for SSP with linear function approximation.
5. L OWER B OUND
We also provide a hardness result for learning linear mixture SSP by proving the lower bound for the expected regret suffered by any deterministic learning algorithms.
Theorem 5.5.
Under Assumption 3.1, suppose d ≥ 2 , B (cid:63) ≥ 2 and K > ( d − 1) 2 / 2 12 .
Then for any possibly non-stationary history-dependent policy π , there exists a linear mixture SSP instance with parameter θ ∗ such that
[Equation]
Remark 5.6.
The expectation in (5.2) is over the trajectories induced by executing the policy π in the SSP environment parameterized by θ ∗ .
Note that here we allow the policy π to be non-stationary and history-dependent.
This is equivalent to assuming a deterministic learning algorithm, which is sufﬁcient for establishing a lower bound (Cohen et al., 2020).
Remark 5.7.
Our instance for the lower bound can be also adapted to a linear SSP instance (Vial et al., 2021), which yields a Ω( dB (cid:63) √ K ) lower bound.
(See Remark D.1 for a detailed discussion.)
P ROOF S KETCH OF THE M AIN R ESULTS
In this section, we give a proof sketch of the main results in Section 5.
Due to space limit, we defer the proof of the lemmas to the appendix.
6. P ROOF OF T HEOREM 5.
In this subsection, we prove Theorem 5.2, which gives the regret upper bound of Algorithm 1 for positive cost functions.
The proof relies on the following intermediate result.
Theorem 6.1.
Under Assumption 3.1 and 3.2, for any δ > 0 , let ρ = 0 and β t =
B (cid:112) d log (4( t 2 + t 3 B 2 /λ ) /δ )
+ √ λd for some B ≥ B (cid:63) where λ ≥ 1 and ρ = 0 .
Then with probability at least 1 − δ , the regret of Algorithm 1 satisﬁes
[Equation]
where T is the total number of steps.
Remark 6.2.
Theorem 6.1 gives an (cid:101) O ( √ T ) regret upper bound with respect to the total number of steps T .
However, for SSP problems, the horizon of each episode is unknown and T can be far greater than K .
Thus, Theorem 6.1 is not satisfactory due to its dependence on T , which is unknown and random.
To deal with this problem, we further prove Theorem 5.2, which translates the dependence on T into the dependence on K .
Theorem 6.1 applies to the general cost function with ρ set to 0.
Note that the regret upper bound depends on the total number of time steps T , which is random.
To replace the T -dependence by the
K -dependence, it sufﬁces to show that T =
(cid:101)
O ( K ) .
As mentioned in Section 5.1, this can be easily derived under Assumption 5.1.
We are now ready to prove Theorem 5.2.
Proof of Theorem 5.2.
The total cost in K episodes is upper bound by R K + KB (cid:63) and is lower bounded by T · c min .
Together with Theorem 6.1, with probability at least 1 − δ , we have
T · c min ≤ 6 β T (cid:115)
[Equation]
(cid:18)
[Equation]
Solving the above inequality for the total number of steps T , we obtain that
[Equation]
Plugging this into Theorem 6.1 yields the desired result.
Note that for the general cost functions, by simply picking ρ = K − 1 / 3 the result immediately follows from the case of positive costs, which is summarized in Theorem 5.4.
6. P ROOF S KETCH OF T HEOREM 6.
The main steps in proving Theorem 6.1 include an analysis of EVI and a regret decomposition.
The complete proof can be found in Appendix C.
Analysis of EVI .
By the algorithmic design we elaborated in Section 4, EVI guarantees optimism and ﬁnite-time convergence, which is summarized in Lemma 6.3 below.
Lemma 6.3.
Let ρ = 0 and β t = B (cid:112) d log (4( t 2 + t 3 B 2 /λ ) /δ )
+ √ λd for all t ≥ 1 , where
B ≥ B (cid:63) .
Then with probability at least 1 − δ/ 2 , for all j ≥ 1 , EVI converges in ﬁnite time and the following holds
[Equation]
Note that in Lemma 6.3 the optimism only holds for the EVI output, i.e., V j for any j ≥ 1 .
The initialization V 0 in Line 2 of the main Algorithm 1 does not necessarily satisfy the optimism since it is possible that V (cid:63) ( s ) < 1 for some s .
Still, such an initialization guarantees (cid:107) V 0 (cid:107) ∞
= 1 ≤ B (cid:63) , which is crucial for establishing the optimism for j ≥ 1 .
The proof of Lemma 6.3 is given in Appendix E.1.
Regret Decomposition.
In our analysis, instead of dealing with (3.3) directly, we ﬁrst implicitly decompose the times steps into intervals, which are indexed by m = 1 , . . .
, M in Lemma 6.4 below.
The basic idea here is to decompose all the time steps into disjoint intervals of which the end points are either the end of an episode or the time steps when the EVI subroutine is triggered 3 .
The purpose of such a regret decomposition is to guarantee that within each interval the optimistic action-value function remains the same, so the induced policy.
This is a necessary and common requirement and can be found in the case of discounted inﬁnite horizon MDPs (Zhou et al., 2021b).
Similar decomposition trick has also been used in existing works on SSP (Rosenberg et al., 2020; Rosenberg & Mansour, 2020; Tarbouriech et al., 2021b).
Lemma 6.4.
Assume the event in Lemma 6.3 holds, then we have the following upper bound for the regret deﬁned in (3.3) 4 :
[Equation]
Bounding E 1 and
E
We bound the terms E 1 and E 2 separately.
Note that E 2 is the sum of a martingale difference sequence, and can be bounded by O ( (cid:112) T log( T/δ )) using standard concentration.
Bounding E 1 is more technical and it requires almost all the properties of our algorithmic design.
In detail, we need to show that every time when EVI is triggered, it can output an optimistic action-value function estimator with high probability (by Lemma 6.3).
Second, we need to bound the total difference between the estimated functions and the optimal action-value function.
This follows from the elliptical potential lemma and the determinant-based doubling criterion.
Third, we need to bound the length of the epochs (i.e., the number of time steps between two EVI s), which is achieved by the time-step doubling criterion as explained in Section 4.
C ONCLUSIONS
In this paper, we proposed a novel algorithm for linear mixture SSP and proved its regret upper and lower bounds.
For future work, there are several important directions.
First, there is a B 0 .
5 (cid:63) gap between the current upper and lower bounds.
We believe this gap can be closed by using a
Bernstein-type of conﬁdence set (Zhou et al., 2021a).
Second, it remains open to prove a (cid:101) O ( √ K ) regret bound for linear mixture SSPs for general cost functions when c min = 0 .
R ( M ) is the same as R K .
We use a different notation to emphasize the interval decomposition.
E THICS STATEMENT
We don’t see any potential ethical issues in our work.