A R EDUCTION -B ASED F RAMEWORK FOR C ONSERVATIVE B ANDITS AND R EINFORCEMENT L EARNING
A BSTRACT
We study bandits and reinforcement learning (RL) subject to a conservative constraint where the agent is asked to perform at least as well as a given baseline policy.
This setting is particular relevant in real-world domains including digital marketing, healthcare, production, ﬁnance, etc.
In this paper, we present a reduction-based framework for conservative bandits and RL, in which our core technique is to calculate the necessary and sufﬁcient budget obtained from running the baseline policy.
For lower bounds, we improve the existing lower bound for conservative multi-armed bandits and obtain new lower bounds for conservative linear bandits, tabular RL and low-rank MDP, through a black-box reduction that turns a certain lower bound in the nonconservative setting into a new lower bound in the conservative setting.
For upper bounds, in multi-armed bandits, linear bandits and tabular
RL, our new upper bounds tighten or match existing ones with signiﬁcantly simpler analyses.
We also obtain a new upper bound for conservative low-rank MDP.
I NTRODUCTION
This paper studies online sequential decision making problems such as bandits and reinforcement learning (RL) subject to a conservative constraint.
Speciﬁcally, the agent is given a reliable baseline policy that may not be optimal but still satisfactory.
In conservative bandits and RL, the agent is asked to perform nearly as well (or better) as the baseline policy at all time.
This setting is a natural formalization of many real-world problems such as digital marketing, healthcare, ﬁnance, etc.
For example, a company may want to explore new strategies to maximize proﬁt while simultaneously maintaining proﬁt above a ﬁxed baseline at any time, in order not to be bankrupted.
See (Wu et al., 2016) for more discussions on the motivation of the conservative constraint.
Analogously to the non-conservative case, conservative bandit/RL problems also require us to balance exploration and exploitation carefully.
Meanwhile, to ensure the obtained policies outperform the baseline policy, we need to provide a tractable approach to keep the exploration not too aggressive.
Solving these two problems simultaneously is the key challenge in conservative bandits and RL.
Existing work proposed algorithms for different settings, including bandits (Wu et al., 2016; Kazerouni et al., 2016; Garcelon et al., 2020b; Katariya et al., 2019; Zhang et al., 2019; Du et al., 2020; Wang et al., 2021) and tabular RL (Garcelon et al., 2020a).
However, lower bound exists only for the multi- armed bandit (MAB) setting (Wu et al., 2016), and there is no lower bound for other widely-adopted settings, such as linear bandits, tabular Markov Decision Process (MDP) and low-rank MDP.
In
Section 1.3, we provide a more detailed discussion of the related work.
For each of the different settings considered in the literature (i.e., multi-armed bandits, linear bandits, tabular MDPs), existing approaches rely on ad-hoc algorithm design and analysis of the trade-off between the setting-speciﬁc regret analysis and the conservative constraint.
Furthermore, it is hard to argue about the optimality of the proposed algorithms because it would require clever constructions of the hard instances to prove the non-trivial regret lower bounds under the conservative constraint.
1. O UR C ONTRIBUTIONS
In this paper, we address these limitations and make signiﬁcant progress in studying the general problem of online sequential decision-making with conservative constraint.
We propose a uniﬁed framework that is generally applicable to online sequential decision-making problems.
The common theme underlying our framework is to calculate the necessary and sufﬁcient budget required to enable non-conservative exploration.
Such a budget is obtained by running the baseline policy (cf.
Section 3).
With the new framework, we obtain a novel upper bound on tabular MDPs, which improves the previous result.
And we prove a new upper bound on low-rank MDPs.
Also, we derive the ﬁrst lower bounds for linear bandits, tabular and low-rank MDPs, which shows that our upper bound is tight.
Lower Bounds.
For any speciﬁc problem (e.g., multi-armed bandits, linear bandits), our framework immediately turns a minimax lower bound of the non-conservative setting to a non-trivial lower bound for the conservative case (cf.
Section 4).
We list some examples to showcase the power of our framework for lower bounds.
Full results are given in Table 1.
• We derive a novel lower bound for multi-armed bandits that works on a wider range of parameters than the one derived in (Wu et al., 2016).
In particular, our lower bound shows a more reﬁned dependence on the value of the baseline policy.
• We derive the ﬁrst regret lower bound for conservative exploration in linear bandits , tabular MDPs and low-rank MDPs .
These results allow to establish or disprove the optimality of the algorithms currently available in the literature.
We emphasize our technique for deriving lower bounds is simple and generic, so we believe it can be used to obtain lower bounds for other problems as well.
Upper Bounds.
Our novel view of conservative exploration can also be used to derive high probability regret upper-bounds.
When the suboptimality gap ∆ 0 and the expected return µ 0 of the baseline policy are known, we show that the Budget-Exploration algorithm (Alg. 1) attains minimax optimal regret in a wide variety of sequential decision-making problems, when associated to any minimax optimal non-conservative algorithm speciﬁc to the problem at hand.
In the more realistic (and challenging) scenario where ∆ 0 and µ 0 are unknown, we show how to simply convert an entire class of algorithms with a sublinear non-conservative regret bound into a conservative algorithms with a sublinear regret bound.
We obtain the following results, full details are given in Table 1.
• In the MAB setting, we obtain a regret upper-bound that matches our reﬁned lower-bound, thus improving on existing analysis.
In the linear bandit setting, we match existing bounds that are already minimax optimal.
In the RL setting, we provide two novel results.
First, we provide the ﬁrst minimax optimal result for tabular MDPs, improving over (Garcelon et al., 2020a).
Second, we derive the ﬁrst upper bound for conservative exploration in low-rank MDPs.
Our bound matches the rate of existing non-conservative algorithms though it is not minimax optimal.
How to achieve minimax optimality in low rank MDPs is an open question even in non-conservative exploration.
Again, our reduction technique is simple and generic, and can be used to obtain new results in previously unstudied settings, like
we
did for low-rank MDPs.
1. M AIN D IFFICULTIES AND T ECHNIQUE O VERVIEW 1.2. L OWER B OUNDS
The only lower bound for conservative exploration is by Wu et al.
(2016) who followed a classical approach in the bandit literature.
They constructed a class of hard environments and used an
[Table]
the suboptimality gap of the baseline policy, A the number of actions (or arms), S the number of states and d the feature dimension.
The upper bounds hold both in the case ∆ 0 and µ 0 are unknown since the lack of knowledge changes the regret only by a constant multiplicative factor (cf.
Section 5).
information-theoretic argument to prove the lower bound.
Construction of hard environments is highly non-trivial because one needs to incorporate the hardness from the conservative constraint.
It is also non-trivial to generalize Wu et al.
(2016)’s lower bound to other settings such as conservative linear bandits and RL because one will need new constructions of hard environments for different settings.
We note that new constructions are needed even for non-conservative settings, because simply embedding the hard instances of MAB to other settings cannot give the tightest lower bounds.
See, e.g., Chapter 24 of Lattimore & Szepesvári (2020) and Domingues et al. (2021).
In this paper, We use a completely different approach.
Our key insights are 1) relating the necessary budget to the regret lower bounds of non-conservative sequential decision-making problems , and 2) obtaining sharp lower bounds in the conservative settings via maximizing a quadratic function (cf.
Equation (6) ) .
Comparing with the classical approach, our approach is simpler and more general: ours does not need problem-speciﬁc constructions and can automatically transform any lower bound in a non-conservative problem to the corresponding conservative problem.
See Sectionfor details.
1.2. U PPER B OUNDS
Improvement over Wu et al.
(2016) when ∆ 0 is known.
When ∆ 0 is known, Wu et al.
(2016) proposed an algorithm (BudgetFirst) which ﬁrst plays the baseline policy for enough times and then plays an non-conservative MAB algorithm.
However, their regret bound is not tight because their analysis on the required budget is loose: they accumulate enough budget to play T - step exploration where T is the total number of rounds.
Our main technical insight to obtain the tight regret bound is a sharp analysis on the required budget: by relating the minimax regret upper bounds of UCB algorithms, we show the required budget can be independent of T .
See Section 5 and F for details.
Sharp upper bounds with unknown ∆ 0 .
When ∆ 0 is unknown, the paper by Wu et al.
(2016), its follow-up papers (Kazerouni et al., 2016; Garcelon et al., 2020b; Zhang et al., 2019; Garcelon et al., 2020a), and our paper, all adopt the same algorithmic template: 1) build an online estimate on the lower bound performance of each possible exploration policy, and 2) based on the estimated lower bounds, choose an exploration policy or play the baseline policy.
The key difference and the most non-trivial part in different papers is how to analyze T
(the number of times of executing the baseline policy).
Existing works upper bound T 0 by relating it to the decision criterion for whether to choose the baseline policy or not.
Since for different problem settings, the criteria have different forms, existing papers adopt different problem-speciﬁc analyses, and in some settings, the analyses are not tight (e.g., MAB and tabular RL).
Our analysis approach is different from existing ones: we bound T 0 via maximizing a quadratic function that depends on the minimax regret bounds of non-conservative algorithms and the conservative constraint .
See Section 5 for more details.
1. R ELATED W ORK
Non-conservative exploration has been widely studied in bandits, and minimax optimal algorithms have been provided for the settings considered in this paper (e.g. Lattimore & Szepesvári, 2020).
The exploration problem has been widely studied also in RL but minimax optimal algorithms have not been provided for all the settings.
For any ﬁnite-horizon time-inhomogeneous MDP with S states, A actions and horizon H , the minimax regret lower bound is Ω( √ H 3 SAT ) (Domingues et al., 2021), where T denotes the number of episodes.
For any time-inhomogeneous low-rank MDP with d -dimensional linear representation, the lower-bound is Ω( √ d 2 H 3 T ) (Zhou et al., 2020, Remark 5.8).
While several minimax optimal algorithms have been provided for tabular MDPs (e.g. Azar et al., 2017; Zanette & Brunskill, 2019;
Zhang et al., 2020a;b; Ménard et al., 2021), the gap between upper bound and lower bound is still open in low-rank MDPs, where LSVI-UCB (Jin et al., 2020) attains a (cid:101) O ( √ d 3 H 4 T ) , while ELEANOR (Zanette et al., 2020) improves to (cid:101) O ( √ d 2 H 4 T ) .
In conservative exploration, previous works focus on designing speciﬁc conservative algorithms for different settings.
This conservative scenario was studied in multi-armed bandits (Wu et al., 2016), contextual linear bandits (Kazerouni et al., 2016; Garcelon et al., 2020b), contextual combinatorial bandits (Zhang et al., 2019) and tabular MDPs (Garcelon et al., 2020a).
All these works focused on providing an upper-bound to the regret of a conservative algorithm.
Other problems that have been considered in conservative exploration are combinatorial semi-bandit with exchangeable actions (Katariya et al., 2019) and contextual combinatorial cascading bandits (Wang et al., 2021).
Du et al.
(2020) have recently considered conservative exploration with sample-path constraint.
Our work is also related to safe bandits/RL (Amani et al., 2019; Pacchiano et al., 2021;
Amani et al., 2021) and constrained RL (Altman, 1999; Efroni et al., 2020; Ding et al., 2020; 2021; Chen et al., 2020).
The setting of safe bandits/RL is different from conservative bandits/RL.
Speciﬁcally, the safety constraint requires that the expected cost at each stage is below a certain threshold.
This constraint is stage-wise, and is independent of the history.
On the contrary, the conservative constraint requires that the total reward is not too small.
For the constrained MDP, the goal is to maximize the expected reward value subject to a constraint on the expected utility value (value function with respect to another reward function).
In conservative RL, however, the agnet aims to maximize the expected reward value subject to the constaint that the (same) reward value is not signiﬁcantly worse that of the baseline policy.
P RELIMINARIES
The objective of this section is to provide a uniﬁed view of the settings considered in this paper, i.e., multi-armed bandits, linear bandits, tabular Markov Decision Processes (MDPs) and low-rank MDPs.
We use the RL formalism which encompasses the bandit settings.
Notations.
We begin by introducing some basic notation.
We use ∆( · ) to represent the set of all probability distributions on a set.
For n ∈ N + , we denote [ n ] = { 1 , 2 , . . . , n } .
We use O ( · ) , Θ( · ) , Ω( · ) to denote the big-O, big-Theta, big-Omega notations.
We use (cid:101) O ( · ) to hide logarithmic factors.
We denote A (cid:38) ( (cid:46) )
B if there exists a positive constant c such that A ≥ ( ≤ ) cB .
Tabular MDPs.
A tabular ﬁnite-horizon time-inhomogeneous MDP can be represent as a tuple
M = ( S , A , H, { p h } Hh =1 , s 1 , { r h } Hh =1 ) , where S is the state space, A is the action space, H is the length of each episode and s 1 is the initial state.
At each stage h , every state-action pair ( s, a ) is characterized by a reward distribution with mean r h ( s, a ) and support in [0 , r max ] , and a transition distribution p h ( ·| s, a ) over next states.
We denote by S = |S| and A = |A| .
A (randomized) policy π ∈ Π is a set of functions { π h : S (cid:55)→ ∆( A ) } h ∈
[ H ] .
For each stage h ∈
[ H ] and any state-action pair ( s, a ) ∈ S × A , the value functions of a policy π are deﬁned as:
[Equation]
For each policy π , we deﬁne V πH +1 ( s ) = 0 and Q πH +1 ( s, a ) =
for all s ∈ S , a ∈ A .
There exists an optimal policy π (cid:63) such that Q (cid:63)h ( s, a ) =
Q π (cid:63) h ( s, a ) = max π Q πh ( s, a ) satisfy the optimal Bellman equations Q (cid:63)h ( s, a )
=
r h ( s, a ) + E s (cid:48) ∼ p
h
( s,a )
[ V (cid:63)h +1 ( s (cid:48) )] and V (cid:63)h = max a ∈A { Q (cid:63)h ( s, a ) } .
Then the optimal policy is the greedy policy π (cid:63)h ( s ) =
arg max a ∈A { Q (cid:63)h ( s, a ) } .
Low-Rank MDPs.
We assume that S , A are measurable spaces with possibly inﬁnite number of elements.
For algorithmic tractability, we shall restrict the attention to A being a ﬁnite set with cardinality A .
When the state space is large or uncountable, value functions cannot be represented in tabular form.
A standard approach is to use a parametric representation.
Here, we assume that transitions and rewards are linearly representable (Jin et al., 2020).
Assumption 1 (Low-rank MDP) .
An MDP ( S , A , H, p, r ) is a linear MDP with a feature map φ : S×
A → R d , if for any h ∈ [ H ] , there exist d unknown (signed) measures µ h = (cid:16) µ (1) h , . .
. , µ ( d ) h (cid:17) over S and an unknown vector θ h ∈ R d , such that for any ( x, a ) ∈ S × A , we have
[Equation]
Without loss of generality, we assume (cid:107) φ ( x, a ) (cid:107) ≤ 1 for all ( x, a ) ∈ S × A , and max {(cid:107)
µ h ( S ) (cid:107) , (cid:107) θ h (cid:107)} ≤ √ d for all h ∈
[ H ] .
Under certain technical conditions (e.g., Shreve & Bertsekas, 1978), all the properties of tabular
MDPs extend to low-rank MDPs.
In addition, the state-action value function of any policy π is linearly representable in low-rank MDPs.
Formally, for any policy π and stage h ∈ [ H ] , there exists θ πh ∈ R d such that Q πh ( s, a ) =
(cid:104) φ ( s, a ) , θ πh (cid:105) .
Connection between RL and Bandits.
To have a uniﬁed view, we can represent a multi-armed bandit as a tabular MDP with S = 1 , A actions, H = 1 and self-loop transitions in s 1 .
In multiarmed bandits, we consider only deterministic policies so
that Π = A , then V π ( s 1 )
=
r ( s 1 , π ( s 1 )) and the optimal policy is simply π (cid:63) =
arg max a ∈A r ( s 1 , a ) .
Similarly, a linear bandit can be modeled through low-rank MDPs with H = 1 .
For generality, we allow the action space to be possibly uncounted and we deﬁne the value of a deterministic policy π = a ( Π = A ) as V π 1 ( s 1 ) =
r 1 ( s 1 , a ) =
(cid:104) φ ( s 1 , a ) , θ 1 (cid:105) .
The optimal policy π (cid:63) is thus such that π (cid:63) =
arg max a ∈A (cid:104) φ ( s 1 , a ) , θ 1 (cid:105) .
We refer the reader to Appendix A for details.
G ENERAL F RAMEWORK F OR C ONSERVATIVE E XPLORATION
With the uniﬁed view provided in the previous section, we can consider a generic sequential decisionmaking problem P over T ∈ N (cid:63) episodes.
We consider the standard online interaction protocol where, at each episode t ∈ [ T ] , the learning agent A selects a policy π t , observes and stores a trajectory ( s i , a i , r i ) i ∈
[ H ] , updates the policy and restart with the next episode.
We evaluate the performance of the learner through the pseudo-regret .
Let V π
=
V π 1 ( s 1 ) be the value function of a policy π , then the regret is deﬁned as:
[Equation]
In conservative exploration, the learner aims to minimize the regret while guaranteeing that, at any episode t , their expected performance is (nearly) above the one of a baseline policy π 0 .
Formally, given a possibly randomized baseline policy π 0 ∈ Π and a conservative level α ∈ [0 , 1] , the learner should satisfy w.h.p.
that
[Equation]
We assume that the value of conservative policy V π 0 is known to the agent.
Such assumption can be seen in previous works such as Wu et al.
; Kazerouni et al.
; Garcelon et al.
(2020b;a).
This assumption is reasonable in practice because usually the baseline policy has been used for a long time and is well-characterized, and its value can be estimated using historical data.
Even if we do not know the value of baseline policy, we can estimate it during the algorithm (e.g., Section 3.5 in Wu et al.
and we omit here for simplicity.
3. B UDGET OF A C ONSERVATIVE A LGORITHM
Given the set of policies { π t } t ∈ [ T ] selected by a conservative algorithm A , we can divide the episodes into the set T
= { t ≤ T | π t = π 0 } and its complement T c 0 = { t ≤ T | π t (cid:54) =
π 0 } =
[ T ] \ T 0 .
The set T c 0 denotes the episodes where the algorithm played an exploratory policy, i.e., it had enough budget to satisfy condition (3) through a policy π l (cid:54) =
π 0 .
This sequence of non-baseline policies { π t } t ∈T c 0 deﬁnes a new algorithm (cid:101) A , that we refer as the non-conservative algorithm.
However, the algorithm A is conservative therefore, for any δ > 0 and t ∈ [ T ] , we have with probability at least 1 − δ that (cid:80) tl
=1 V π l ≥ (1 − α ) tV π 0 .
Hence, for any t ∈ [ T ] we have:
[Equation]
where T 0 ,t = T 0 ∩ [ t ] and T c 0 ,t
= T c 0 ∩ [ t ] .
Taking maximum over t in Eq.
(4), we have that with high probability the conservative algorithm A is such that
[Equation]
The quantity on the right of the previous equation is exactly the amount of reward that the conservative algorithm A has to collect by playing the baseline policy.
Hence this quantity acts as a conservative budget B .
The higher it is, the more A needs to play the baseline policy to satisfy the conservative condition.
In other words, it is the least amount of reward that an algorithm needs to not violate the conservative constraint.
We now extend this notion to any (non necessarily conservative) algorithm.
Deﬁnition 1.
For any T ∈ N (cid:63) , set of episodes O ⊂ [ T ] and arbitrary sequence of policies { π t } t ∈O , the budget of this sequence of policies is deﬁned as:
[Equation]
R EGRET L OWER B OUND FOR C ONSERVATIVE E XPLORATION
In this section, we leverage the framework introduced in Section 3 to build lower bounds for several problems.
Our result is based on the notion of budget deﬁned in Section 3.
This notion is used to build an algorithm whose regret is a lower bound for any conservative algorithm.
Theorem 1 (Conservative Exploration Regret Lower Bound) .
Let’s consider a decision-making problem P over T steps, a conservative level α ∈ [0 , 1] , a baseline policy π 0 , an algorithm A and δ ∈ (0 , 1) .
We assume that:
• Lower-bound for non-conservative exploration.
There exists a ξ ∈ R + and T 0 ∈ N such that for any algorithm A (cid:48)
there exists an environment (instance of P ) such that with probability at least 1 − δ , R T ( P , A (cid:48) ) ≥ ξ √ T for T ≥ T 0 .
• A is conservative.
The algorithm A is conservative, that is to say with probability at least 1 − δ for
any
[Equation]
Then, there exists an environment (instance of problem P ) and T 0 ∈ N such that with probability at least 1 − δ and T ≥ T 0 :
[Equation]
where ∆ 0 = V (cid:63) − V π 0 is the sub-optimality gap of policy π 0 .
Theorem 1 provides a general framework deriving lower-bounds for conservative exploration and highlights the impact of the baseline policy on the regret.
In particular, it shows that in any sequential decision-making problem, after a sufﬁciently large number of episodes the conservative condition can be veriﬁed and the baseline policy has no impact anymore on the learning process.
The only requirement is the knowledge of a lower-bound for the non-conservative case.
Before instantiating the result in speciﬁc settings, we provide an intuition about how this result is derived and what is the role of the conservative budget B .
Proof Sketch.
Let us consider a conservative algorithm A
= { π t | t ≤ T } , which is associated to a non-conservative algorithm (cid:101)
A = { π t | t ∈ T c 0 } with T c 0 and T 0 the set of non-conservative and conservative episodes as deﬁned in Sec.
Now if E |T 0 | ≥
[Equation]
) (i.e. the algorithm plays π 0 too many times), then the regret caused by π 0 is at least
[Equation]
When E |T 0 | <
[Equation]
αV
[Equation]
+
[Equation]
) , consider the budget of T 0 deﬁned in Deﬁnition 1:
[Equation]
where E (cid:104) R T c
A ( A c ) ( t ) (cid:105) is the regret incurred by the rounds in T c 0 .
Now if E (cid:104) R T c
A ( A c ) ( t ) (cid:105) ≥ ξ √ t , we have B T c
( A c ) (cid:38)
ξ 2 αV π 0 +∆ 0 by taking maximum on the right handside of (6) (viewing RHS as a quadratic function of √ t ).
Therefore E |T 0 | ≥ B T c
[Equation]
A
[Equation]
) and the regret is also no smaller than
[Equation] [Equation]
0 ) , which completes the proof.
Example of Lower Bounds.
For instance, in the multi-armed bandits, by leveraging the lower-bound in (Thm. 15 .
2 Lattimore & Szepesvári, 2020), we can obtain the following corollary of Theorem 1.
This result is more general than the lower bound in Wu et al.
(2016) where they have a restriction that ∆αµ 0 +∆ 0 ≥ 0 .
See Appendix E for details.
Corollary 1.
For any K ∈ N (cid:63) , α ∈ [0 , 1] , µ 0 ∈ [0 , 1] , δ ∈ (0 , 1) and a conservative algorithm A then there exists µ ∈ [0 , 1] K such that (cid:80) tl =1 µ π l ≥ (1 − α ) µ 0 t with high probability for any t ≤ T .
Then, for T ≥
[Equation]
The generality of Theorem 1 allows us to derive lower-bounds for conservative exploration in many different problems, where the lower-bound was unknown.
Table 1 reports the lower-bound obtained through Theorem 1.
Please refer to Appendix B for lower-bounds for non-conservative exploration.
In linear bandits, the lower bound we obtain matches the result in (Kazerouni et al., 2016; Garcelon et al., 2020b), showing the optimality of their algorithms.
In tabular MDPs, our result shows that the dependence on S, A and H of CUCBVI (Garcelon et al., 2020a) is not optimal.
Finally, by instantiating Theorem 1 in low-rank MDPs, we obtain the ﬁrst lower bound for this setting.
U PPER B OUNDS
In this section, we show how to leverage the framework of Sec.
3 to derive an algorithm for any conservative sequential decision-making problem.
We ﬁrst show that when knowing ∆ 0 a simple algorithm achieves a minimax regret, as prescribed by our lower bound of Sec. 4.
Then, we show how to remove this knowledge without hurting the performance by combining our framework and the idea of lower conﬁdence bound.
5. T HE B UDGET -E XPORATION A LGORITHM
Given a non-conservative algorithm (cid:101) A , the minimum amount of rewards needed to play this nonconservative algorithm for T consecutive steps is the budget deﬁned in Def. 1.
Indeed, if we denote by { ˜ π l | l ≤ T } the sequence of non-conservative policies executed by (cid:101) A , then for any set
O ⊂ [ T ] the budget can be rewritten as:
[Equation]
Input: A non-conservative algorithm (cid:101)
A , conservative policy cumulative reward V π 0 , conservative level: α
) ,baseline action gap: ∆
=
V
V π 0 and a constant C π
∆ 0 and
Play
; else
Play according to
; end 8 end = max
[Equation]
(cid:88) l ∈O∩ [ t ] (cid:16)
[Equation]
(cid:12)(cid:12)
[Equation]
(cid:12)(cid:12)(cid:17) .
Let’s deﬁne R O∩
[ t ] ( (cid:101) A ) :
= (cid:80) l ∈O∩ [ t ] V (cid:63) − V π l the regret over the time steps in O of the nonconservative algorithm (cid:101) A .
For most non-conservative algorithms with minimax regret bound, ˜ R T ( (cid:101) A , O ) =
O ( C (cid:112) |O ∩ [ t ] | ) w.h.p., where C ∈ R is a problem-dependent quantity as in Theorem 1.
For example, in multi-armed bandit C = √ A for the UCB algorithm or C = √ H 3 SA for the UCBVIBF algorithm (Azar et al., 2017).
This implies that the budget required by (cid:101)
A is at least
[Equation]
Therefore, the simple algorithm playing the baseline policy for the ﬁrst T 0 :
= O (
[Equation]
) steps and then running the non-conservative algorithm (cid:101) A , is conservative.
We call such algorithm
Budget-Exporation (see Alg. 1).
This algorithm is conservative and minimax optimal.
Indeed, we can show (see Theorem 2) that the regret upper bound of Budget-Exporation matches the lower bounds of Section 4.
While knowing ∆ 0 in advance may be a restrictive assumption, it is interesting that a two-stage algorithm structure (deploying a baseline policy and then a non-conservative policy) is enough to achieve minimax optimality.
Theorem 2.
Consider an algorithm (cid:101) A , δ ∈ (0 , 1) and constant C ∈ R such that with probability at least 1 − δ , for any T ≥ 1 , R T ( (cid:101) A ) ≤ (cid:101)
O ( C √ T ) .
Then for any T ≥ 1 , the regret of BudgetExporation is bounded with probability at least 1 − δ by (cid:101)
O ( C √ T
+
[Equation]
Instantiating Thm.
1 with (cid:101)
A being the UCB algorithm
(Lattimore & Szepesvári, 2020), then C = √ A and the regret of Budget-Exporation is bounded w.h.p.
by (cid:101) O ( √ AT +
[Equation]
) , that matches our novel lower bound introduced in Sec. 4.
Similar results can be obtained for the other settings, see Table 1.
In linear bandit we consider LinUCB as the non-conservative algorithm, leading to
C = d .
Similarly, in tabular MDP and low-ran MDPs, we get C = √ H 3 SA and C = √
d
3 Hrespectively using UCBVI-BF (Azar et al., 2017) and LSVI-UCB (Jin et al., 2020).
Refer to Tablefor a complete comparison of the results.
5. T HE LCBCE A LGORITHM
When ∆ 0 is unknown, we aim to use the same idea as Budget-Exporation , that is to say to play a policy different than the baseline one only if the budget is positive.
To achieve this, we need to build an online estimate of the conservative budget which amounts to build a lower conﬁdence bound (w.h.p.) on the value function of any policy π .
Therefore, assuming a non-conservative algorithm (cid:101)
A builds such conﬁdence bounds, for example by estimate the MDP as done by Garcelon et al.
(2020a), we show how our budget framework helps to derive a conservative regret bound.
Let’s consider a non-conservative algorithm (cid:101)
A = { π t | t ≤ T } able to construct a high probability lower bound on the set of selected policies.
That is, for any time t ≤ T and δ ∈ (0 , 1) , (cid:101)
A computes a sequence of real numbers ( λ π k t ( δ ))
k ≤ t such that with probability at least 1 − δ , for all k ≤ t , λ π k t ( δ ) ≤ V π k .
Using these lower bounds, we can deﬁne a proxy to the budget for (cid:101) B T,δ
( O , (cid:101) A ) for
Input: A non-conservative algorithm (cid:101) A , δ ∈ (0 , 1) , lower conﬁdence bounds λ π k t ≤ V π k , conservative policy value V π 0 , α ∈ (0 , 1) 1 Set B = 0 ; // the accumulated budget 2 Set t (cid:48)
= 0 ; // the number of steps in which the agent acts as (cid:101)
A 3 for t = 1 , 2 , ..., T do (cid:101)
A gives lower bound λ t (cid:48)
+1 and a policy ˜ π t (cid:48) +1 ;
Set λ
= (cid:80) t (cid:48) k
=1 λ ˜
π k t (cid:48)
+1 + λ ˜ π t (cid:48)
+1 t (cid:48) +1 ; // lower of total
if λ − ( t (cid:48) + 1) αV π 0 < B then
Play π t = π 0 and set B = B + αV π 0 ; else
Play π t =
˜
π t (cid:48)
+1 and set t (cid:48)
=
t (cid:48)
+ 1 ; end 11 end any subset O ⊂ [ T ] by
[Equation]
(7) with ( π l ) l ∈
O the sequence of policies computed by the non-conservative algorithm (cid:101) A .
Then following from the deﬁnition of ( λ π l t ( δ ))
l ≤ t , we have that with probability at least 1 − δ that (cid:101) B T,δ (cid:16) O, (cid:101)
A (cid:17) ≥ B T ( O , (cid:101) A ) .
This shows that it is possible to compute (cid:101) B T,δ (cid:16) O,
(cid:101) A (cid:17) without knowledge of the environment and the baseline parameters.
The idea of our algorithm is now to play a non-conservative policy π t at time t only if the difference between the proxy to the budget of (cid:101)
A and the reward accumulated by playing the baseline policy is negative.
Formally, the condition is (cid:101) B t,δ (cid:16) S t ∪
[Equation]
, (cid:101) A (cid:17) ≤ αV
π
( t − 1 − | S t | ) where S t is the set of time step where a non-conservative policy was deployed in episodes before t .
As a result, the minimum budget that (cid:101)
A requires to be conservative is max
[Equation] [Equation]
The algorithm, called Lower Conﬁdence Bound for Conservative Exploration (LCBCE), is detailed in Alg. 2.
Next, we show the regret bound of LCBCE .
The proof is in Appendix D.
Theorem 3.
Consider an algorithm (cid:101) A , δ ∈ (0 , 1) and constant C ∈ R such that with probability at least 1 − δ , for any T ≥ 1 , R T ( (cid:101) A ) ≤ (cid:101)
O ( C √ T ) .
If (cid:101)
A computes lower conﬁdence bound such that (cid:80) t k =1 (cid:0) V π k − λ π k t (cid:1) ≤ (cid:101) O
( C √ T )
with probability at least 1 − δ , then for any T ≥ 1 , the regret of
LCBCE is bounded with probability at least 1 − δ by (cid:101)
O ( C √ T
+
[Equation]
In the MAB and tabular case, LCBCE paired with UCB achieves a better regret bound compared with previous papers(Garcelon et al., 2020a; Wu et al., 2016).
We also provide the ﬁrst minimax optimal bound for the case of unknown baseline parameters.
Finally, in low rank MDPs we recover the same rate as in the case of known baseline.
See Table 1.
C ONCLUSION
We present a uniﬁed framework for conservative exploration in sequential decision-making problems.
This framework can be leveraged to derive both minimax lower and upper bounds.
In bandits, we provide novel lower bounds that highlighted the optimality of existing algorithms.
In RL, we provide the ﬁrst lower bound for tabular MDPs and a matching upper bounds, and the ﬁrst analysis for low rank MDPs.
An interesting question is whether one can leverage this framework to derive problemdependent logarithmic bounds for conservative exploration.
Another direction is to investigate model-free algorithms (e.g., Q-learning (Jin et al., 2018)) for conservative exploration.
A CKNOWLEDGEMENTS
This work was supported by National Key R&D Program of China (2018YFB1402600), Exploratory
Research Project of Zhejiang Lab (No. 2022RC0AN02), BJNSF (L172037).
Project 2020BD supported by PKUBaidu Fund.