Provably Feedback-Efficient Reinforcement Learning via Active Reward Learning
Anonymous Author(s)
AffiliationAddress
Abstract
An appropriate reward function is of paramount importance in specifying a taskin reinforcement learning (RL).
Yet, it is known to be extremely challenging inpractice to design a correct reward function for even simple tasks.
Human-in-theloop (HiL) RL allows humans to communicate complex goals to the RL agent byproviding various types of feedback.
However, despite achieving great empiricalsuccesses, HiL RL usually requires too much feedback from a human teacher andalso suffers from insufficient theoretical understanding.
In this paper, we focus onaddressing this issue from a theoretical perspective, aiming to provide provablyfeedback-efficient algorithmic frameworks that take human-in-the-loop to specifyrewards of given tasks.
We provide an active-learning -based RL algorithm thatfirst explores the environment without specifying a reward function and then asksa human teacher for only a few queries about the rewards of a task at some stateaction pairs.
After that, the algorithm guarantees to provide a nearly optimal policyfor the task with high probability.
We show that, even with the presence of randomnoise in the feedback, the algorithm only takes (cid:101)
O
( H dim 2 R ) queries on the rewardfunction to provide an ε -optimal policy for any ε > 0 .
Here H is the horizonof the RL environment, and dim R specifies the complexity of the function classrepresenting the reward function.
In contrast, standard RL algorithms require toquery the reward function for at least Ω(poly
( d, 1 /ε )) state-action pairs where ddepends on the complexity of the environmental transition.
Introduction
A suitable reward function is essential for specifying a reinforcement learning (RL) agent to performa complex task.
Yet obvious approaches such as hand-designed reward is not scalable for largenumber of tasks, especially in the multitask settings [Wilson et al., 2007, Brunskill and Li, 2013,
Yu et al., 2020, Sodhani et al., 2021] and can also be extremely challenging (e g., [Ng et al., 1999,
Marthi, 2007] shows even intuitive reward shaping can lead to undesired side effects).
Recently, apopular framework called Human-in-the-loop (HiL) RL [Knox and Stone, 2009, Christiano et al.
,2017, MacGlashan et al., 2017, Ibarz et al., 2018, et al., 2021, Wang et al., 2022] gains moreinterests as it allows humans to communicate complex goals to the RL agent directly by providingvarious types of feedback.
In this sense, a reward function can be learned automatically and can alsobe corrected at proper times if unwanted behavior is happening.
Despite its promising empirical performance, HiL algorithms still suffer from insufficient theoretical understanding and possess drawbacks Arakawa et al.
[2018], e.g., it assumes humans can give precise numerical rewards and do so without delay and at every time step, which are usually not true.
Moreover, these approaches usually train on every new task separately and cannot incorporate exiting experiences
In this paper, we attempt to address the above issues of incorporating humans’ feedback in RL from atheoretical perspective.
In particular, we would like to address (1) the high feedback complexity issue– i.e., the algorithms in practice usually require large amount feedback from humans to be accurate;(2) feedback from humans can be noisy and non-numerical; (3) in need for support of multiple tasks.
In particular we consider a fixed unknown RL environment, and formulate a task as an unknown butfixed reward function.
A human who wants the agent to accomplish the task needs to communicatethe reward to the agent.
It is not possible to directly specify the parameters of the reward function asthe human may not know it exactly as well, but is able to specify good actions at any given state.
Tocapture the non-numerical feedback issue, we assume that the feedback we can get for an action isonly binary – whether an action is “good” or “bad”.
We further assume that the feedback is noisyin the sense that the feedback is only correct with certain probability.
Lastly, we require that thealgorithm, after some initial exploration phase, should be able to accomplish multiple tasks by onlyquerying the reward rather than the environment again.
In the supervised learning setting, if we only aim to learn a reward function, the feedback complexitycan be well-addressed by the active learning framework Settles [2009], Hanneke et al.
[2014] – analgorithm only queries a few samples of the reward entries and then provide a good estimator.
Yetthis become challenging in the RL setting as it is a sequential decision making problem – state-actionpairs that are important in the supervised learning setting may not be accessible in the RL setting.
Therefore, to apply similar ideas in RL, we need a way to explore the environment and collect samplesthat are important for reward learning.
Fortunately, there were a number of recent works focusing“reward-free” exploration Jin et al.
[2020a], Wang et al.
[2020a] on the environment.
Hence, applyingsuch an algorithm would not affect the feedback complexity.
Additionally it is possible for us to reusethe collected data for multiple tasks.
Our proposed theoretical framework is a non-trivial integration of reward-free reinforcement learningand active learning.
The algorithm possesses two phases: in phase I, it performs reward-free RL toexplore the environment and collect the small but necessary amount of the information about theenvironment; in phase II, the algorithm performs active learning to query the human for the reward atonly a few state-action pairs and then provide a near optimal policy for the tasks with high probability.
The algorithm is guaranteed to work even the feedback is noisy and binary and can solve multipletasks in phase II.
Below we summarize our contributions:
We propose a theoretical framework for incorporating humans’ feedback in RL.
The framework contains two phases: an unsupervised exploration and an active reward learning phase.
Since the two phases are separated, our framework is suitable for multi-task RL.2.
Our framework deals with a general and realistic case where the human feedback is stochasticand binary-i.e., we only ask the human teacher to specify whether an action is “good” or“bad”.
We design an efficient active learning algorithm for learning the reward function fromthis kind of feedback.3.
Our query complexity is minimal because it is independent of both the environmentalcomplexity d and target policy accuracy ε .
In contrast, standard RL algorithms require querythe reward function for at least Ω(poly
( d, 1 /ε )) state-action pairs.
Thus our work providesa theoretical validation for the recent empirical HiL RL works where the number of queriesis significantly smaller than the number of environmental steps.4.
Moreover, we shows the efficacy of our framework in the offline RL setting, where theenvironmental transition dataset is given beforehand.
Related Work
Our active reward learning algorithm is inspired by a line of works [Cesa-Bianchi et al., 2009, Dekelet al., 2010, Agarwal, 2013] considering online classification problems.
Our reward-free explorationalgorithm is most relevant to Wu et al.
[2021], Wang et al.
[2020a] studying reward-free RL intabular/linear setting.
Due to space limit, more related works are deferred to Appendix E.
Preliminaries3.1 Episodic Markov Decision Process
In this paper, we consider the finite-horizon Markov decision process (MDP) M =
( S , A , P, r, H, s 1 ) ,where S is the state space, A is the action space, P
=
{ P h } Hh =1 where P h : S ×
A → △ ( S ) are thetransition operators, r = { r h } Hh =1 where r h : S
×
A → { 0 , 1 } are the deterministic binary rewardfunctions, and H is the planning horizon.
Without loss of generality, we assume that the initial state s 1is fixed.
1 In RL, an agent interacts with the environment episodically.
Each episode consists of H timesteps.
A deterministic policy π chooses an action a ∈ A based on the current state s ∈ S at each timestep h ∈
[ H ] .
Formally, π = { π h } Hh =1 where for each h ∈
[ H ] , π h : S
→ A maps a given state to anaction.
In each episode, the policy π induces a trajectory s 1 , a 1 , r 1 , s 2 , a 2 , r 2 , ..., s H , a H , r H ,
s H
+1where s 1 is fixed, a 1 = π 1 ( s 1 ) , r 1 = r 1 ( s 1 , a 1 ) , s 2 ∼ P 1 ( ·| s 1 , a 1 ) , a 2 =
π 2 ( s 2 ) , etc.
We use Q-function and V-function to evaluate the long-term expected cumulative reward in termsof the current state (state-action pair), and the policy deployed.
Concretely, the Q-function and
V-function are defined as: Q πh ( s, a ) = E (cid:2) (cid:80)
Hh ′ =
h r h ′
( s h ′ , a h ′ ) | s h
= s, a h
= a, π (cid:3) and V πh ( s ) =
E (cid:2) (cid:80)
Hh ′ =
h r h ′
( s h ′ , a h ′ ) | s h
= s, π (cid:3) .
We denote the optimal policy as π ∗ =
{ π ∗ h } h ∈ [ H ] , optimalvalues as Q ∗ h ( s, a ) and V ∗ h ( s ) .
Sometimes it is convenient to consider the Q-function and V-functionwhere the true reward function is replaced by a estimated one ˆ r = {
ˆ r h } h ∈
[ H ] .
We denote themas Q πh ( s, a, ˆ r ) and V πh ( s, ˆ r ) .
We also denote the corresponding optimal policy and value as π ∗ (ˆ r ) ,
Q ∗ h ( s, a, ˆ r ) and V ∗ h ( s, ˆ r ) .
Additional Notations.
We define the infinity-norm of function f : S ×
A → R as ∥ f ∥ ∞ =sup ( s,a ) ∈S×A | f
( s, a ) | .
For a set of state-action pairs Z ⊆ S
× A and a function f : S ×
A → R ,we define ∥ f ∥
Z = (cid:16)(cid:80) ( s,a ) ∈Z f
( s, a ) 2 (cid:17) 1 / 2 .
Technical Overview
In this section we give a overview of our learning scenario and notations, as well as the maintechniques.
The learning process divides into two phases.
4.1 Phase 1: Unsupervised Exploration
The first step is to explore the environment without reward signals.
Then we can query the humanteacher about the reward function in the explored region.
We adopt the reward-free explorationtechnique developed in Jin et al.
[2020a], Wang et al.
[2020a].
The agent is encouraged to doexploration by maximizing the cumulative exploration bonus .
Concretely, we gather K trajectories
D = { ( s kh , a kh ) } ( h,k ) ∈
[ H ] ×
[ K ] by interacting with the environment.
We can strategically choose whichpolicy to use.
At the beginning of the k -th episode, we calculate a policy π k based on the history ofthe first k − 1 episodes and use π k to induce a trajectory { ( s kh , a kh ) } h ∈ [ H ] .
A similar approach called unsupervised pre-training [Sharma et al., 2020, Liu and Abbeel, 2021] hasbeen successfully used in practice.
Concretely, in unsupervised pre-training agents are encouraged todo exploration by maximizing various intrinsic rewards , such as prediction errors [Houthooft et al.,2016] and count-based state-novelty [Tang et al., 2017].
4.2 Phase 2: Active Reward Learning
The second step is to learn a proper reward function from human feedback.
Our work assumes thatthe underlying valid reward is 1-0 binary, which is interpreted as good action and bad action.
Weremark that RL problems with binary rewards represent a large group of RL problems that are suitableand relatively easy for having human-in-the-loop.
A representative group of problems is the binaryjudgments: For example, suppose we want a robot to learn to do a backflip.
A human teacher willjudge whether a flip is successful and assign a reward of 1 for success and 0 for failure.
Furthermore, our framework can also be generalized to RL problems with n -uniform discrete rewards.
The detaileddiscussion is defered to Appendix F.2 due to space limit.
Concretely, consider a fixed stage
h ∈
[ H ] , and we are trying to learn r h from the human response.
Each time we can query a datum z
=
( s, a ) and receive an independent random response Y ∈ { 0 , 1 }from the human expert, with distribution:
[Equation]
Here f ∗ h is the human response model and needs to be learned from data.
We assume that theunderlying valid reward of z can be determined by f ∗ h ( z ) in the following manner:
[Equation]
Note that the query returns 1 with a probability greater than 12 if and only if the underlying validreward is 1.
To make the number of queries as small as possible, we choose a small subset ofinformative data to query the human.
We adopt ideas in the pool-based active learning literature andselect informative queries greedily.
We show that only (cid:101) O
( H dim 2 R ) queries need to be answered bythe human teacher.
The active query method is widely used in human-involved reward learning inpractice and shows superior performance than uniform sampling Christiano et al.
[2017], Ibarz et al.[2018], Lee et al.
After we learn a proper reward function ˆ r , we use past experience D and ˆ r to plan for a good policy.
Note that in this phase we are not allowed for further interaction with the environment.
In themulti-task RL setting, we can run Phase 2 for multiple times and reuse the data collected in Phase 1.
Now we discuss the efficacy of our framework.
A naive approach for reward learning via humanfeedback is asking the human teacher to evaluate the reward function in each round.
This approachresults in equal environmental steps and number of queries.
This high query frequency is unacceptablefor large-scale problems.
For example, in Lee et al.
[2021] the agent learns complex tasks with veryfew queries ( ∼ 10 2 to 10 3 queries) to the human compared to the number of environmental steps( ∼ 10 6 steps) by utilizing active query technique.
From the theoretical perspective, usual RL samplecomplexity bound scales with ∝ poly( d, 1 ε ) , where d is the complexity measure of the environmentaltransition and ε is the target policy accuracy.
This quantity can be huge when the environment iscomplex (i.e., d is large) or with small target accuracy.
Our query complexity is desirable since it isindependent of both d and 1 /ε .
Pool-Based Active Reward Learning
In this section we formally introduce our algorithm for active reward learning.
We consider a fixedstage h and learn r h by querying a small subset of Z h
= { ( s kh , a kh ) } k ∈
[ K ] .
We omit the subscripth in this section, i.e., we use Z , z k , r, f ∗ to denote Z h , z kh , r h , f ∗ h in this section.
Since Z is givenbefore the learning process starts, we refer to this learning scenario as pool-based active learning.
Our purpose is to learn a reward function
ˆ r ( · ) such that r ( z ) =
ˆ r ( z ) for most of z in Z .
At the sametime, we hope the number of queries can be as small as possible.
We assume F is a pre-specified function class to learn f ∗ from, and F is known as a prior.
Weassume that F has enough expressive power to represent the human response.
Concretely, we assumethe following realizability .
Assumption 1 (Realizability) .
f ∗ ∈ F .
The learning problem can be arbitrarily difficult, especially when f ∗ ( z ) is close to 12 , in which case itwill be difficult to determine the true value of r ( z ) .
To give a problem-dependent bound, we assumethe following bounded noise assumption.
In the literature on statistical learning, this assumption isalso referred to as Massart noise [Massart and Nédélec, 2006, Giné and Koltchinskii, 2006, Hannekeet al., 2014].
Our framework can also work under the low noise assumption - due to space limit, wedefer the discussion to Appendix F.3.
Assumption 2 (Bounded Noise) .
There exists ∆ > 0 , such that for all z ∈ S × A , | f ∗ ( z ) − 12 | > ∆ .
The value of the margin ∆ depends on the intrinsic difficulty of the reward learning problem and thecapacity of the human teacher.
For example, if the reward is rather easy to specify and the human teacher is a field expert, and can always give the right answer with a probability of at least 80%, then∆ will be 0.3.
But if the learning problem is hard or the human teacher is unfamiliar with the problemand can only give near-random answers, then ∆ will be very small.
But in that case, we won’t hopethe human teacher can help us in the first place.
So a typical good value for ∆ should be a constant.
Examples We give two examples of F that is frequently studied in the active learning literature.
Inthe linear model, the function class F consists of f in the following form: f ( z ) =
⟨ ϕ ( z ) ,w ⟩ +1.
In thelogistic model, the function class F consists of f in the following form: f ( z ) = exp ⟨ ϕ ( z ) ,w ⟩ 1+exp ⟨ ϕ
( z ) ,w ⟩ .
Hereϕ : S ×
A → R d is a fixed and known feature extractor, and w ∈ R d .
The complexity of F essentially depends on the learning complexity of the human response model.
We use the following Eluder dimension [Russo and Van Roy, 2014] to characterize the complexity of
F .
The eluder dimension serves as a common complexity measure of a general non-linear functionclass in both reinforcement learning literature [Osband and Van Roy, 2014, Ayoub et al., 2020, Wanget al., 2020c, Jin et al., 2021a] and active learning literature [Chen et al., 2021].
Definition 1 (Eluder Dimension) .
Let ε ≥ 0 and Z
= { ( s i , a i ) } ni
=1 ⊆ S ×
A be a sequence ofstate-action pairs.
(1) A state-action pair ( s, a ) ∈ S
× A is ε -dependent on Z with respect to F if any f, f ′ ∈
Fsatisfying ∥
f − f ′
∥
Z ≤ ε also satisfies | f ( s, a )
− f ′ ( s, a ) | ≤ ε .(2)
An ( s, a ) is ε -independent of Z with respect to F if ( s, a ) is not ε -dependent on Z .(3)
The ε -eluder
dimension dim E ( F , ε ) of a function class F is the length of the longest sequence ofelements in S ×
A such that, for some ε ′ ≥ ε , every element is ε ′ -independent of its predecessors.
(4) The eluder dimension of a function class F is defined as dim E ( F ) :
=
lim sup α ↓ 0 dim E ( F ,α ) log(1 /α ) .
We remark that a wide range of function classes, including linear functions, generalized linearfunctions and bounded degree polynomials, have bounded eluder dimension.
Definition 2 (Covering Number and Kolmogorov Dimension) .
For any ε > 0 , there exists anε -cover C
( F , ε ) ⊆
F with size |C ( F , ε ) | ≤ N ( F , ε ) , such that for any f ∈ F , there
existsf ′ ∈ C ( F , ε ) with ∥
f − f ′ ∥ ∞ ≤ ε .
The Kolmogorov dimension of F is defined as: dim K ( F ) :=
lim sup α ↓ 0 log( N ( F ,α )) log(1 /α ) .
The Kolmogorov dimension is also bounded by O ( d ) for linear/generalized linear function class.
Throughout this paper, we denote dim( F ) :
= max { dim E ( F ) , dim K ( F ) } as the complexity measureof F .
When F is the class of d -dimensional linear/generalized linear functions, dim( F ) is boundedby O ( d ) .
5.1 Algorithm
We describe our algorithm for learning the human response model and the underlying reward function.
We sequentially choose which data points to query.
Denote Z k the first k points that we decide toquery and initial Z 0 to be an empty set.
For each z ∈ Z , we use the following bonus function tomeasure the information gain of querying z , i.e., how much new information z contains compared to
Z k − 1 :
[Equation]
We then simply choose z k to be arg max z ∈Z b k ( z ) .
After the N query points are determined, wequery their labels from a human.
The human response model is then learned by solving a least-squaresregression:
[Equation]
The human response model is used for estimating the underlying reward function.
We round (cid:101) f tothe cover C ( F , ∆ / 2) to ensure that there are a finite number of possibilities of such functions – thisgives us the convenience of applying union bound in our analysis.
Indeed, we believe a more refinedanalysis would remove the requirement of rounding but will make the analysis much more involved.
The whole algorithm is presented in Algorithm 1.
Note that such an interactive mode with the humanteacher is non-adaptive since all queries are given to the human teacher in one batch.
This propertymakes our algorithm desirable in practice.
Here we assume the value of ∆ is known as a prior.
Wecan extend our results to the case where ∆ is unknown.
Due to space limit, we defer the discussion to
Appendix F.1.
Input: Data Pool Z = { z i } i ∈
[ T ] , margin ∆ , failure probability δ ∈ (0 , 1)
Z 0 ← {} //Query
Dataset
Set N ← C 1 · (dim 2 ( F )+dim( F ) · log(1 /δ ))
· (log 2 (dim( F )))
∆for k = 1 , 2 , ..., N do β ← C 2 · (cid:112) log(1 /δ )
+ log N · dim( F )
Set the bonus function: b k ( · ) ← sup f,f ′ ∈F , ∥ f − f ′ ∥
Z k − 1 ≤ β | f ( · ) − f ′ ( · ) | z k ← arg max z ∈Z b k ( z )
Z k ← Z k − 1 ∪ { z k } end for for z ∈ Z N do
Ask the human expert for a label l ( z ) ∈ { 0 , 1 } end for
Estimate the human model as (cid:101) f = arg min f ∈F (cid:80) z ∈Z N ( f ( z ) − l ( z ))
Let ˆ f ∈ C ( F , ∆ / 2) such that ∥ ˆ f − (cid:101) f ∥ ∞ ≤ ∆ /
Estimate the underlying true reward: ˆ r ( · ) =
(cid:40) 1 , ˆ f ( · ) > 1 /0 , ˆ f ( · ) ≤ 1 /return:
The estimated reward function ˆ r .
5.2 Theoretical Guarantee
Theorem 1.
With probability at least 1 − δ , for all z ∈ Z , we have, ˆ r ( z ) =
r ( z ) .
The total numberof queries is bounded by O (cid:16) (dim 2 ( F )+dim( F ) · log(1 /δ ))
· (log 2 (dim( F )))
∆(cid:17) .
Proof Sketch The first step is to show that the sum of bonus functions (cid:80)
Kk
=
b k ( z k ) is boundedby O ( d √ K ) using ideas in Russo and Van Roy [2014].
Note that the bonus function b k ( · ) is nonincreasing.
Thus we can show that after selecting N = (cid:101) O
( d 2 ∆ 2 ) points, for all z ∈ Z , the bonusfunction of z does not exceed ∆ .
By the bounded-noise assumption, we know that the reward labelfor z is correct for all z .
Online RL with Active Reward Learning
In this section we consider how to apply active reward learning method in the online RL setting.
Inthis setting the agent is allowed to actively explore the environment without reward signal in theexploration phase.
We consider both tabular MDP and linear MDP cases.
6.1 Linear MDP with Positive Features
The linear MDP assumption was first studied in Yang and Wang [2019] and then applied in the onlinesetting Jin et al.
[2020b].
It is assumed that the agent is given a feature extractor ϕ : S ×
A → R dand the transition model can be predicted by linear functions of the give feature extractor.
In ourwork, we additionally assume that the coordinates of ϕ are all positive .
This assumption essentiallyreduce the linear MDP model to the soft state aggregation model [Singh et al., 1994, Duan et al.
As will be seen later, the latent state structure helps the learned reward function to generalize.
Assumption 3 (Linear MDP with Non-Negative Features) .
For all h ∈ [ H ] , we assume thatthere exists a function µ h : S → R d such that P h ( s ′ | s, a ) =
⟨ µ h ( s ′ ) , ϕ ( s, a ) ⟩ .
Moreover, for all( s, a ) ∈ S × A , the coordinates of ϕ ( s, a ) and µ h ( s, a ) are all non-negative.
6.2 Exploration Phase
In the exploration phase, inspired by former works on reward-free RL, we use optimistic least-squaresvalue iteration (LSVI) based algorithm with zero reward.
In the linear case, for any V : S → R , we estimate P h V in the following manner
[Equation]
w ←
[Equation]
For the tabular case, we simply use the empirical estimation of P h :
[Equation]
and define (cid:98) P kh V in the conventional manner.
Here N kh
( s, a, s ′ )
= (cid:80) k − 1 τ =1 1 { ( s τh , a τh , s τh +1 ) =(
s, a, s ′ ) } and N kh ( s, a ) =
(cid:80)
k − 1 τ =1 1 { ( s τh , a τh ) =
( s, a ) } are the numbers of visit time.
The following optimism bonus Γ kh ( · , · ) is sufficient to guarantee optimism in standard regret minimization RL algorithms.
(The choices of β tbl and β lin is specified in the appendix)
[Equation]
In our setting we enlarge the optimism bonus to the following exploration bonus .
[Equation]
Case) .
[Equation]
We then set the optimistic Q-function as
[Equation]
and define the exploration policy as the greedy policy with respect to Q kh .
Algorithm 2 UCBVI-Exploration for k = 1 , 2 , ..., K do
V kH +1 ← 0 , Q kH +1 ←for h
= H, H − 1 , ..., 1 do
Estimate (cid:98) P
kh
V
kh
+
( · , · ) using (1) or (2)
Set the optimism bonus Γ kh ( · , · ) using (3)
Set the exploration bonus b kh ( · , · ) using (4).
Set the optimistic Q-function Q kh ( · , · ) ← Π [0 ,H − h +1]
[ (cid:98) P kh V kh +1 ( · , · ) + b kh ( · , · )] π kh ( · ) ← arg max a ∈A Q kh ( · , a )
V kh ( · ) ← max a ∈A Q kh ( · , a ) end for
Execute policy π k = { π k h } h ∈
[ H ] to induce a trajectory s k 1 , a k 1 , ..., s k H , a k H , s k H +1 .
end for return: Dataset D = { ( s kh , a kh ) }
( h,k ) ∈
[ H ] ×
[ K ]
6.3 Reward Learning & Planning Phase
After the exploration phase, we run the active reward learning algorithm introduced before on thecollected dataset.
In the linear setting, we replace the original action with uniform random action.
Wethen use the learned reward function to plan for a near-optimal policy.
We still add optimism bonusto guarantee optimism.
The whole algorithm is presented in Algorithm
6.4 Theoretical Guarantee
Theorem 2.
In the linear case, our algorithm can find an ε -optimal policy with probability at least1 − δ , with at most
O
[Equation]
Algorithm
Input: Dataset D = { ( s kh , a kh ) } ( h,k ) ∈
[ H ] ×
[ K ] for h = 1 , 2 , ..., H do if Linear Case then (cid:101)
Z h ← { ( s kh , (cid:101) a kh ) } k ∈ [ K ] , where { (cid:101) a kh } k ∈
[ K ] are sampled i.i.d.
from Unif( A )
ˆ r h ← Active Reward Learning ( (cid:101)
Z h , ∆ , δ/ (2 H )) .
else if Tabular Case then
Z h ← { ( s kh , a kh ) } k ∈
[ K ] ˆ r h ← Active Reward Learning ( Z h , ∆ , δ/ (2 H )) .
end if end for for k = 1 , 2 , ...,
K do
V kH +1 ← 0 , Q kH +1 ← 0 for h = H, H − 1 , ..., 1 do
Estimate (cid:98) P
kh
V
kh
+
( · , · ) using (1) or (2)
Set the optimism bonus Γ kh ( · , · ) using (3)
Set the optimistic Q-function Q kh ( · , · ) ← Π [0 ,H − h +1]
[ˆ r h ( · , · ) + (cid:98) P
kh V kh +
+ Γ kh ( · , · )]
ˆ π kh ( · ) ← arg max a ∈A Q kh ( · , a )
V kh ( · ) ← max a ∈A Q kh ( · , a ) end for end for return: ˆ π drawn uniformly from { ˆ π k } Kk =1 where
ˆ
π
k
=
ˆ π kh } h ∈ [ H ] episodes.
In the tabular case, our algorithm can find an ε -optimal policy with probability at least1 − δ , with at most
[Equation]
) episodes.
In both cases, the total number of queries to the reward is bounded by (cid:101)
O
( H · dim 2 ( F ) / ∆ 2 ) .
Remark 1.
Theorem 2 readily extends to multi-task RL setting by replacing δ with δ/N and applyinga union bound over all tasks, where N is the number of tasks.
The corresponding sample complexitybound only increase by a factor of poly log( N ) .
Remark 2.
Standard RL algorithms require to query the reward function for at leastΩ( max { dim R , dim P } 2 ε) times ( dim R and dim P stand for the complexity of the reward/transition function).
Compared to their bound, our feedback complexity bound has two merits: 1)
In practice thetransition function is generally more complex than the reward function, thus max { dim R , dim P } ≫dim R ; 2)
Our bound is independent of ε - note that ε can be arbitrarily small, whereas ∆ is aconstant.
Proof Sketch The suboptimality of the policy ˆ π can be decomposed into two parts:
[Equation]
where (i) correspond to the planning error in the planning phase (Algorithm 3) and (ii) correspondto the estimation error of the reward ˆ r .
By standard techniques from the reward-free RL, (i) canbe upper bounded by the expected summation of the exploration bonuses in the exploration phase.
In order to bound (ii), we need the learned reward function to be universally correct, not just onthe explored region.
We show that the dataset collected in the exploration phase essentially coverthe state space (tabular case) or the latent state space.
Since the reward function class has boundedcomplexity
Offline RL with Active Reward Learning
In this section we consider the offline RL setting, where the dataset D is provided beforehand.
Weshow that our active reward learning algorithm can still work well in this setting.
In order to givemeaningful result, we assume the following compliance property of D with respect to the underlying
MDP.
This assumption is firstly introduced in Jin et al.
[2021b].
Unlike many literature for offline
RL, we do not require strong coverage assumptions, e.g., concentratability [Szepesvári and Munos,2005, Antos et al., 2008, Chen and Jiang, 2019].
Definition 3 (Compliance) .
For a dataset D = { ( s kh , a kh ) } ( h,k ) ∈
[ H ] ×
[ K ] , let P D be the joint dis- 295 tribution of the data collecting process.
We say D is compliant with the underlying MDP if
P D ( s kh +
=
s |{ ( s jh , a jh ) } kj =1 , { s jh +1 } k − 1 j =1 ) =
P h ( s | s kh , a kh ) holds for all h ∈
[ H ] , k ∈
[ K ] , s ∈ S .
7.1 Algorithm and Theoretical Guarantee
At the beginning of the algorithm we call the active reward learning algorithm to estimate the rewardfunction.
Inspired by Jin et al.
[2021b], we estimated the optimal Q-value Q k using pessimistic valueiteration with empirical transition and learned reward function.
The policy is defined as the greedypolicy with respect to Q k .
The full algorithm and theoretical guarantee is stated below.
Input: Dataset D = { ( s kh , a kh ) } ( h,k ) ∈
H ] ×
K for
H do
Z end
1 , ..., 1 do Γ h ( · , · ) ← β ′ tbl · ( N h ( · , · ) + 1) − 1 /
Q h ( · , · ) ← Π [0 ,H − h +1]
[ˆ r h ( · , · ) + (cid:98) P h (cid:98) V
h
+1 ( · , · ) − 2Γ h ( · , · )]
ˆ π h ( · ) ← arg max a ∈A Q h ( · , a )
V h ( · ) ← max a ∈A Q h ( · , a ) end for return: ˆ π = { ˆ π h } h ∈
[ H ]
Theorem 3.
With probability at least 1 − δ , the sub-optimal gap of ˆ π is bounded by
[Equation]
And the total number of queries is bounded by (cid:101)
O ( H · dim 2 ( F ) / ∆ 2 ) .
The proof of Theorem 3 is deferred to the appendix.
Conclusions and Discussions
In this work, we provide a provably feedback-efficient algorithmic framework that takes human-in-theloop to specify rewards of given tasks.
Our proposed framework theoretically addresses several issuesof incorporating humans’ feedback in RL, such as noisy, non-numerical feedback and high feedbackcomplexity.
Technically, our work integrates reward-free RL and active learning in a non-trivialway.
The current framework is limited to information gain-based active learning, and an interestingfuture direction is incorporating different active learning methods, such as disagreement-based activelearning, into our framework.
From a broad perspective, our work is a theoretical validation of recent empirical successes in HiL
RL.
Our results also brings new ideas to practice: it provides a new type of selection criterion that canbe used in active queries; it suggests that one can use recently developed reward-free RL algorithmsfor unsupervised pre-training.
These ideas can be combined with existing deep RL frameworks to bescalable.
A limitation of the current work is that it mainly focus on theory, and we leave the empiricaltest of these ideas in real-world deep RL as future work.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope
[Yes
(b) Did you describe the limitations of your work
[Yes See Section 8 for future directions. (c)
Did you discuss any potential negative societal impacts of your work
[N/A
] Thiswork is theoretical in nature and does not have immediate social impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Assump- tions 1,2,
(b) Did you include complete proofs of all theoretical results?
[Yes] Proof of all theoremsand lemmas are included in the Supplementary Material.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes] The source
code is included in the supplementary material.
One may run Figure1.m and Figure2.m
to reproduce the results in Figure 1 and Figure 2, respectively.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] Please refer to Appendix A.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[No] We did not report the error bars due to time limit.
We
will provide more detailed experiments in the final version.
(d) Did you include the total amount of compute and the type of resources used (e.g.,
type of GPUs, internal cluster, or cloud provider)?
[N/A]
The amount of compute is
negligible since the environment is very small.
Our results can be easily reproduced in
a personal laptop.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Please refer to
Appendix A.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
Errata Several (cid:98) P kh V kh in the main paper should be (
cid:98) P kh V
kh
+1 .
Some full stops are missing in themain paper.
We have fixed these typos in the current main paper.
Road map for the appendices
In Section A we provide the numerical simulation results.
From
Section B to Section D, we give proofs of Theorem 1 to Theorem 3.
A Numerical Simulations
We run a few experiments to test the efficacy of our algorithmic framework and verify our theory.
Wesimulate a random two-stage ( H = 2 ) tabular MDP with S = 20 , A = 10 (stage 1) or 3 (stage 2).
We consider the linear response model
[Equation]
where ϕ ( z ) , w ∈ R d and d = 5 .
We fix the environmental steps in the exploration phase to be
K = 1000 .
In our environment, the noise margin parameter ∆ (defined in Assumption 2) can beadjusted by removing states that do not satisfy the assumption.
The error is defined as
[Equation]
where ˆ π is the learned policy and s 1 is the fixed initial state.
Active Learning v.s. Passive Learning.
Figure 1 shows a comparison between the algorithm withactive reward learning (our Algorithm 1) and the algorithm with passive learning.
The only differenceis that instead of actively choosing queries, the passive learning algorithm uniformly samples queriesfrom the dataset collected in the exploration phase.
Figure 1 shows that the active reward learningmethod significantly reduces the number of queries needed to achieve a target policy accuracy.
In thisexperiment, the noise margin parameter is ∆ = 0 . 05 .
The Effect of the Noise Margin ∆ .
Theorem 1 suggests that the noise margin ∆ will significantlyinfluence the difficulty of the reward learning problem.
Figure 2 verifies this effect when the numberof queries is fixed to N = 100 .
Moreover, we also compare between active learning and passivelearning in this setting.
Implementation Details.
The transition probabilities of the MDP are generated uniformly from the
S -dimensional probability simplex.
The features in the response model are generated from a uniformball distribution with random scaling.
Results in Figure 1 and Figure 2 are averaged over 100 trials.
We use a few MATLAB package that are listed below.
The source code is given is the supplementarymaterial.
One may run Figure1.m and Figure2.m to reproduce the results in Figure 1 and Figure 2,respectively.
Dahua Lin (2022).
Sampling from a discrete distribution (link), MATLAB Central File Exchange.
Retrieved May 26, 2022.
David (2022).
Uniform Spherical Distribution Generator (link), MATLAB Central File Exchange.
Retrieved May 26, 2022.
Roger Stafford link
Retrieved May 26, 2022.
B Proof of Theorem
The next lemma bound the error of the regression.
Lemma 1.
With probability at least 1 − δ ,
[Equation]
Proof.
For any f ∈ F and z ∈ Z N , considerξ ( z, f ) =
( f ( z ) − f ∗ ( z ))
( f ∗ ( z ) − l ( z )) where l ( z ) is the response from the human expert.
Note that
E [ ξ ( z, f )] = 0 , | ξ ( z, f ) | ≤ 2 | f
( z ) − f ∗ ( z ) | .
By Hoeffding’s inequality, for a fixed f ∈ F , we have that
Pr (cid:34)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (cid:88)
z ∈Z N ξ ( z, f ) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≥ ε (cid:35)
[Equation]
Let
[Equation]
We have that with probability at least 1 − δ , for all f ∈ N ( F , 1 /N ) ,
[Equation]
Condition on the above event for the rest of the proof.
Consider any f ∈ F , there exists g ∈
N ( F , 1 /N ) such that ∥ g − f ∥ ∞ ≤ 1 /N .
Thus we have that
[Equation]
In particular,(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
On the other hand,
[Equation]
Thus we have
[Equation]
which implies
[Equation]
as desired.
Following the analysis in Russo and Van Roy [2014], we can bound the sum of bonuses in terms ofthe eluder dimension of F .
Lemma 2.
[Equation]
Proof.
For k ∈ [ K ] , denote Z k
= { z τ } k − 1 τ =1 .
For any given ε > 0 and h ∈ [ H ] , let L =
z k | k ∈[ N ] , b k ( z k ) > ε } with |L| =
L .
We will show that there exists z k ∈ L such that z k is ε -dependenton at least L/ dim E ( F , ε ) − 1 disjoint subsequences in Z k ∩ L .
Denote N = L/ dim E ( F , ε ) − 1 .
We decompose L into N +1 disjoint subsets, L = ∪ N +1 j =1 L j by the following procedure.
We initialize
L j = {} for all j ∈ [ N + 1] and consider each z k ∈ L sequentially.
For each z k ∈ L , we find thesmallest 1 ≤ j ≤ N such that z k is ε -independent on L j with respect to F .
We set j = N + 1 ifsuch j does not exist.
We add z k into L j afterwards.
When the decomposition of L is finished, L N +1 must be nonempty since L j contains at most dim E ( F , ε ) elements for j ∈ [ N ] .
For any z k ∈ L N +1 ,z k is ε -dependent on at least L/ dim E ( F , ε ) − 1 disjoint subsequences in Z k ∩ L .
On the other hand, there exist f 1 , f 2 ∈ F such that | f 1 ( z k ) − f 2 ( z k ) | > ε and ∥ f 1 − f 2 ∥ 2 Z k ≤ β 2 .
By the definition of ε -dependent we have
[Equation]
which implies
[Equation]
Let b 1 ≥ b 2 ≥ ...
≥ b N be a permutation of { b k ( z k ) } k ∈ [ N ] .
For any b k ≥ 1 /N , we have
[Equation]
which implies
[Equation]
Moreover, we have b k ≤ 1 .
Therefore,
N
[Equation]
log N · β ) as desired.
Proof of Theorem 1.
Note that the preference functions { b k ( · ) } are non-increasing.
Thus by Lemma2, we have that
[Equation]
Substituting the value of N and β , with a proper choice of C 1 , we conclude that
[Equation]
Combining the above result with Lemma 1, we have for all z ∈ Z ,
[Equation]
which further implies
[Equation]
Thus by the hard margin assumption of f ∗ , we complete the proof.
C Proof of Theorem
C.1 Proof for the Linear Case
We choose β lin to be:
[Equation]
Throughout the proof, we denote ϕ kh
= ϕ
( s kh , a kh ) for all ( h, k ) ∈
[ H ] ×
[ K ] .
We further denote
[Equation]
We denote
[Equation]
+
[Equation]
2 +
[Equation]
and thus (cid:98) P kh V kh +1 ( · , · ) =
ϕ ( · , · ) T w kh .
Similarly,
[Equation] [Equation]
and (cid:98) P kh V kh +1 ( · , · ) =
ϕ ( · , · ) T w kh .
We have the following lemma on the norm of w kh and w kh :
Lemma 3.
For all ( h, k ) ∈ [ H ] × [ K ] ,
[Equation]
Proof.
The proof is identical to Lemma B.2 in Jin et al.
[2020b].
C.1.1 Analysis of the Planning Error
Analysis in this section utilizes techniques from Jin et al.
[2020b], Wang et al.
[2020a].
Denote all candidate reward function as R , which contains all functions in the from:r ( · ) =
(cid:26) 1 , f ( · ) > 1 /0 , f ( · ) ≤ 1 /where f ∈ C ( F , ∆ / 2) .
Clearly we have for all h ∈
[ H ] , the estimated reward function ˆ r h ∈ R .
Notethat the size of R is bounded by N ( F , ∆ / 2) .
Now we state the standard concentration bound for the linear MDP firstly introduced in Jin et al.[2020b].
Lemma 4.
With probability at least 1 − δ , for all h ∈
[ H ] and k ∈
[ K ] ,(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
Proof.
Note that the value function V kh +1 is of the form:
[Equation]
where ∥ w ∥ 2 ≤ 2 H √ dK , Λ ⪰ I , and r ∈ R .
Consider a fixed r ∈ R .
Identical to Lemma D.4 of Jin et al.
[2020b], we have that: with probabilityat least 1 − δ , for all V ( · ) in the above form (5) (with that fixed r ),
[Equation]
By replacing δ with δ/ |R| and applying a union bound over all r ∈ R , we have that with probabilityat least 1 − δ , [Equation] ∆
[Equation]
for all V ( · ) in the above form (5) (for all r ∈ R ).
And we are done.
The next lemma bound the single-step planning error.
Lemma 5 (Single-Step Planning Error) .
In Algorithm 2 and Algorithm 3, with probability at least1 − δ , for any h ∈ [ H ] , k ∈
[ K ] and ( s, a ) ∈ S × A ,| (cid:98) P kh V
kh {++
+}
( s, a ) − P
h V
kh {++
+}
( s, a ) | ≤ Γ kh
[Equation]
s, a ) and
[Equation]
Proof.
We provide the proof for the first inequality and that for the second inequality is identical.
Note that
[Equation]
We denote
[Equation]
s
[Equation]
µ
[Equation]
( s
[Equation]
kh {++
+}
( s
[Equation]
) , thus P h V
kh +
( s, a ) =
ϕ ( s, a ) T (cid:101) w kh .
By ∥ µ h ( S ) ∥ 2 ≤ √ d , we have ∥ (cid:101) w kh ∥ 2 ≤ H √ d .
Note thatϕ
[Equation]
Thus
[Equation]
) where the last inequality is obtained by plugging in the bound for ∥ (cid:101) w kh ∥ 2 and(cid:13)(cid:13)(cid:13)(cid:80)
[Equation]
Lemma 4).
The next lemma guarantees optimism in the planning phase.
Lemma 6 (Optimism) .
For Algorithm 3, with probability at least 1 − δ , for any h ∈ [ H +1] , k ∈
[ K ]and ( s, a ) ∈ S × A ,
[Equation]
Proof.
We condition on the event defined in Lemma 5.
The proof is by induction on h .
The result forh = H + 1 clearly holds.
Suppose the result for h + 1 holds.
Note that for all ( s, a ) ∈ S
× A andk ∈
[ K ] ,
[Equation]
In the above proof we assume Q kh ( s, a ) ≤ H − h +1 , since we always have Q ∗ h ( s, a, ˆ r ) ≤
H − h +1 .
Moreover,
[Equation]
and we are done.
The next lemma bound the regret in the planning phase in terms of the expected sum of explorationbonuses.
Lemma 7 (Regret Decomposition) .
With probability at least 1 − δ , for any h ∈ [ H + 1] , k ∈ [ K ] ,s ∈ S ,
[Equation]
Proof.
We prove the lemma by induction on h .
The conclusion clearly holds for h
= H + 1 .
Assumethat the conclusion holds for h + 1 , i.e., for any k ∈ [ K ] and s ∈ S ,
[Equation]
Consider the case for h .
Denote a = ˆ π kh ( s ) =
arg max a ∈A Q kh ( · , a ) for the rest of the proof.
Wehave
[Equation]
[Equation] [Equation]
H − h +1]
[Equation]
ˆ r h
[Equation]
, a
[Equation]
kh
[Equation]
Γ kh
[Equation]
a )] and
[Equation]
Thus we have
[Equation]
as desired.
Lemma 8.
With probability at least 1 − δ ,
[Equation]
Proof.
Note that Algorithm 2 in the linear case is identical to the Algorithm 1 (LSVI-UCB) in Jinet al.
[2020b] with zero reward, except for a enlarged bonus.
(cid:80) K k
=1 V k 1 ( s 1 ) corresponds to the regretand can be estimated using standard techniques.
We omit the proof for brevity.
Lemma 9.
With probability at least 1 − δ ,
[Equation]
K
Proof.
We condition on the event defined in Lemma 7 and Lemma 8.
Note that
[Equation]
We complete the proof by plugging in the bound given in Lemma 8.
C.1.2 Latent State Representation
Our purpose is to show that ˆ r will not incur much error under any policy.
We need to exploit thelatent state structure of the MDP to bound the generalization error of ˆ r .
Firstly we need to derive thelatent state model (a.k.a, soft state aggregation model) from the non-negative feature model.
Note that for all ( s, a ) ∈ S × A , (cid:80) s ′
∈S P h
( s ′ | s, a ) = 1 , thus we have(cid:42)
[Equation]
a
[Equation]
=
[Equation]
Denote µ h := (cid:80) s ′ ∈S µ h ( s ′ ) .
We define a latent state space X
= { 1 , 2 , ..., d } .
Let each state-actionpair induces a posterior distribution over X :ψ h : S ×
A → ∆( X ) , where ψ h ( s, a )[ x ] =
ϕ
( s, a )[ x ] · µ h [ x ] .
Since ⟨ ϕ ( s, a ) , µ h ⟩ = 1 , ψ ( · ) is a probability distribution.
For each latent variable induces a emission distribution over Sν h : X → ∆( S ) , where ν h ( x )[ s ′ ]
= µ h ( s ′ )
[ x ] /µ h
[ x ] .
ν h ( · ) is also a probability distribution by definition.
In stage h we sample x h ∼ ψ ( s h , a h ) ands h +1 ∼ ν h ( x h ) .
The trajectory can be amplified as:s 1 , a 1 , x 1 , s 2 , ..., s H , a H , x H , s H +1 .
It suffice to check the transition probability is maintained:
[Equation]
and we are done.
C.1.3 Analysis of the Reward Error
The error can be decomposed in the following manner.
Lemma 10.
For any policy π , we have that(cid:12)(cid:12)(cid:12)
[Equation]
(cid:12)(cid:12)(cid:12) ≤
H (cid:88) h
=
E π (cid:34)(cid:88) a ∈A
[Equation]
(cid:35)
From now on we fix a stage
h ∈
[ H − 1] , and try to analyze the error of the learned reward functionin stage h + 1 .
We leverage the latent variable structure to analyze the error of ˆ r .
For j ∈ [ d ] , denotec j the number of times we visit the j -th latent state in stage h during K episodes.
[Equation]
We define the error of ˆ r starting from the j -th latent state as:
[Equation]
where ν h denotes the emission probability.
Thus we can further define the error vector of ˆ r as
[Equation]
Denoting ϕ π = E π ϕ
( s h , a h ) .
The next key lemma bound the error induced by the reward function.
The proof of Lemma 11 is defered to the next section.
Lemma 11.
With probability at least 1 − δ , for any policy π ,
[Equation]
Lemma 12.
With probability at least 1 − δ ,
[Equation]
K
Proof.
Similar to Lemma 3.2 of Wang et al.
[2020a], with probability at least 1 − δ , for any policy π ,we have that (we treat Γ Kh ( · , · ) /H as a reward function)
[Equation]
where Γ Kh ( s h , a h ) = min { β lin (cid:113) ϕ ( s h , a h ) T (Λ Kh ) − 1 ϕ
( s h , a h ) , H } .
We condition on this event forthe rest of the proof.
Note that
[Equation]
Thus we have
[Equation]
K
Note that by Jensen’s inequality,
[Equation]
and we are done.
For a distribution λ ∈ ∆( S × A ) , denote the population risk of an estimated reward function ˆ r in the( h + 1) -th stage as
[Equation]
For j ∈ [ d ] , denote λ j the distribution starting from the j -th hidden state and take random action, i.e.,
[Equation]
h
[Equation]
Unif [Equation]
Then the error vector can be represented as
[Equation]
Note that every time we arrive at j -th hidden state, i.e., x kh =
j , it indicates that ( s kh +1 , (cid:101) a kh +1 ) is arandom sample from λ j .
Denote the empirical risk of ˆ r for the first m samples from λ j as err ν j ,m (ˆ r ) .
Classic supervised learning theory gives us the following bound.
Lemma 13.
With probability at least 1 − δ , for all m ∈ [ K ] and reward function r ∈ R consistentwith the first m samples from ν j ,
[Equation]
We conclude that with probability at least 1 − δ , for any policy π ,
H (cid:88)
[Equation]
=
[Equation]
K
By the above results we conclude the following lemma.
Lemma 14.
With probability at least 1 − δ ,
[Equation]
Proof of Theorem 2 in the Linear Case.
Note that
[Equation]
Combing Lemma 9 and Lemma 14 completes the proof.
C.1.4 Proof of Lemma
[-of Lemma 
-]
Denote that
[Equation]
We define an expected version of c j :
[Equation]
The next lemma bound e j in terms of c j .
Lemma 15.
With probability at least 1 − δ , for all j ∈ [ d ] ,
[Equation]
for some absolute constant C > 0 .
Proof of Lemma 11.
Note that
[Equation]
Thus the error caused by ˆ r in stage h can be represented as:
[Equation]
where w ′ [ j ] = w [ j ] · µ h
[ j ]
Here we bound the error vector w ′ under the Λ h -norm.
For i ∈ [ K ] , denote ϕ i = ϕ
( s ih , a ih ) .
Thenwe have that
[Equation]
i =
d (cid:88)
j = ( ψ i [ j ] w [ j ])
[Equation]
+ d ≤ d d (cid:88) j = (cid:32) K (cid:88) i =
ψ i [ j ] w [ j ] (cid:33)+ d ( Note that ϕ i
[ j ] w [ j ] ≥ 0) =
d d (cid:88)
j = ( e j w [ j ]) 2 + d ≲ d d (cid:88) j
= ( c j w [ j ] + log( K/δ ))
+ d ≲ d d (cid:88)
j = ( c j w [ j ]) 2 + d 2 log 2 ( K/δ )
Thus we have that
[Equation]
w [
[Equation]
[Equation]
C.2 Proof of the Tabular Case
We choose β tbl to be:
[Equation]
Before the proof, we remark that directly treating the linear case as a special case of the tabular casewill derive a much looser bound.
Our proof is based on the analysis in Wu et al.
[2021] and Zanetteand Brunskill [2019].
We streamline the key lemmas and omit some of the detailed proofs for brevity.
C.2.1 Good Events
Denoting w kh
( s, a ) =
P
[Equation]
{ ( s h , a h ) =
( s, a ) } , we construct the following “good event”.
[Equation] [Equation]
Lemma 16.
[Equation]
Proof.
The proof is identical to that of Lemma 1 in Wu et al.
We omit it for brevity.
Lemma 17.
If events G P , G (cid:98) P hold, then for all V 1 , V 2 : S → [0 , H ] satisfying V 1 ≤ V 2 and( s, a ) ∈ S ×
A
[Equation]
and
[Equation]
Proof.
The proof is identical to that of Lemma 3 in Wu et al.
We omit it for brevity.
C.2.2 Analysis
Lemma 18 (Optimism of the planning phase) .
If G H holds, for all ( s, a ) ∈ S × A ,
h ∈
[ H ] andk ∈ [ K ] ,
[Equation]
Proof.
Note that the estimated reward function ˆ r h is always true for N kh
( s, a ) > 0 .
On the otherhand, for N kh ( s, a ) = 0 , the optimistic Q-function Q kh ( s, a ) is H − h
+
and the value of ˆ r h will notaffect Q kh ( s, a ) .
The rest of the proof follows from standard techniques from Azar et al.
Lemma 19.
If events G H , G (cid:98) P holds, then for all s ∈ S , h ∈ [ H ] and k ∈
[ K ] ,
[Equation]
In particular,
[Equation]
Proof.
The proof is identical to that of Lemma 10 in Wu et al.
We omit it for brevity.
Lemma 20.
If events G H , G P holds, then for all k ∈
[ K ] ,
[Equation]
Proof.
We denote a 1 = π kh
( s 1 ) .
Note that
[Equation]
and we are done.
Lemma 21.
With probability at least 1 − δ ,
K (cid:88) k
=
V k 1 ( s 1 ) ≲ √ H 4 SAKι + H 3 S 2 Aι, where ι = log( HSAK/δ ) .
Proof.
We set L kh =
{ ( s, a ) | (cid:80) k − 1 τ =1 w τh ( s, a ) ≥ 2 ι } .
Note that
[Equation]
We estimate these parts separately.
By definition we have
[Equation]
Note that
K (cid:88)
[Equation]
and
K (cid:88) k
=
H (cid:88) h
= (cid:88)
[Equation]
K
[Equation]
Combining the above three parts we complete the proof.
Lemma 22.
[Equation]
, where ι = log( HSAK/δ ) .
Proof.
Combining the results in Lemma 17,Lemma 19 and Lemma 21 completes the proof.
Proof of Theorem 2 in the Tabular Case.
Plugging in the value of K into Lemma 22 completes theproof.
D Proof of Theorem
We choose β ′ tbl to be:
[Equation]
By standard techniques developed in Jaksch et al.
[2010], we bound the L1-norm of the estimationerror of (cid:98) P h in the following sense.
Lemma 23.
For τ ∈ [ K ] , h ∈ [ H ] and ( s,
a ) ∈ S × A , denote (cid:98) P τh ( ·| s, a ) ∈ R S the empiricalestimation of P h ( ·| s, a ) based on the first τ samples from ( s, a ) in D .
Then with probability at least1 − δ ,
[Equation]
τ
The next lemma bound the single-step planning error.
Lemma 24.
With probability at least 1 − δ , for all ( s, a ) ∈ S × A and all h ∈
[ H ] ,|
[Equation]
s
[Equation]
s, a
[Equation]
and|
[Equation]
, a
[Equation]
Proof.
Note that
[Equation]
The proof of the first part follows from the results in Lemma 23.
The second part is obvious.
Define the model evaluation error to be
[Equation]
Lemma 25.
Under the event defined in Lemma 24, for all ( s, a ) ∈ S × A and all h ∈ [ H ] , [Equation] a
[Equation]
Proof of Theorem 3.
With Lemma 25, Theorem 3 falls into a special case of Theorem 4.2 of Jin et
al.[2021b].
We omit the complete proof for brevity.
E Related Work
Sample Complexity of Tabular and Linear MDP.
There is a long line of theoretical work on thesample complexity and regret bound for tabular MDP.
See, e.g., [Kearns and Singh, 2002, Jakschet al., 2010, Azar et al., 2017, Jin et al., 2018, Zanette and Brunskill, 2019, Agarwal et al., 2020b,
Wang et al., 2020b, Li et al., 2022].
The linear MDP is first studied in Yang and Wang [2019].
See,e.g., [Yang and Wang, 2020, Jin et al., 2020b, Zanette et al., 2020a, Ayoub et al., 2020, Zhou et al.
,2021a,b] for sample complexity and regret bound for linear MDP.
Unsupervised Exploration for RL.
The reward-free exploration setting is first studied in Jin et al.[2020a].
This setting is later studied under different function approximation scheme: tabular [Kaufmann et al., 2021, Ménard et al., 2021, Wu et al., 2022], linear function approximation [Wang et al.
,2020a, Zanette et al., 2020b, Zhang et al., 2021, Huang et al., 2022, Wagenmaker et al., 2022, Agarwalet al., 2020a, Modi et al., 2021], and general function approximation [Qiu et al., 2021, Kong et al.
,2021, Chen et al., 2022].
Besides, Zhang et al.
[2020], Yin and Wang [2021] study task-agnostic RL,which is a variety of reward-free RL.
Wu et al.
[2021] studies multi-objective RL in the reward-freesetting.
Bai and Jin [2020], Liu et al.
[2021] studies reward-free exploration in Markov games.
Active Learning.
Active learning is relatively well-studied in the context of unsupervised learning.
See, e.g., Dasgupta et al.
[2007], Balcan et al.
[2009], Settles [2009], Hanneke et al.
[2014] and thereferences therein.
Our active reward learning algorithm is inspired by a line of works [Cesa-Bianchiet al., 2009, Dekel et al., 2010, Agarwal, 2013] considering online classification problem where theyassume the response model P ( y | x ) is linear parameterized.
However, their works can not directlyapply to the RL setting and also the non-linear case.
There are also many empirical study-focusedpaper on active reward learning. See, e.g., [Daniel et al., 2015, Christiano et al., 2017, Sadigh et al.
,2017, Bıyık et al., 2019, 2020, Wilde et al., 2020, Lindner et al., 2021, Lee et al., 2021].
Many ofthem share similar algorithmic components with ours, like information gain-based active query andunsupervised pre-training.
But they do not provide finite query complexity bounds.
F ExtensionsF.1 Unknown Noise Margin
In the main paper we assume that the noise margin ∆ is known as a prior, and the algorithms need∆ as a input.
But in reality the value of ∆ is usually unknown to the agent.
Here we provide anapproach to bypass this issue.
We use binary search to guess the value of ∆ .
This only introduces alog factor to the asymptotic sample complexity as we only need to guess logarithmically many times. Algorithm
Input: Data Pool Z = { z i } i ∈ [ T ] , guess margin ∆ , failure probability δ ∈ (0 , 1)
Z 0 ← {} //Query
Dataset
Set N ← C 1 · (dim 2 ( F )+dim( F ) · log(1 /δ ))
· (log 2 (dim( F )))
∆for k = 1 , 2 , ..., N do β ← C 2 · (cid:112) log(1 /δ )
+ log N · dim( F )
Set the bonus function: b k ( · ) ← sup f,f ′ ∈F , ∥ f − f ′ ∥
Z k − 1 ≤ β | f ( · ) − f ′ ( · ) | z k ← arg max z ∈Z b k ( z )
Z k ← Z k − 1 ∪ { z k } end for for z ∈ Z N do
Ask the human expert for a label l ( z ) ∈ { 0 , 1 } end for
Estimate the human model as (cid:101) f = arg min f ∈F (cid:80) z ∈Z N ( f ( z ) − l ( z ))for all z ∈ Z do if | (cid:101) f
( z ) − 1 / 2 | > ∆ / 2 then return false end if end for
Let ˆ f ∈ C ( F , ∆ / 2) such that ∥ ˆ f − (cid:101) f ∥ ∞ ≤ ∆ /
Estimate the underlying true reward: ˆ r ( · ) =
[Equation]
return: The estimated reward function ˆ r .
First, we add a validation step in the active learning algorithm.
After learning the human model (cid:101) f , wetest whether for each data point z in the data pool Z we have (cid:101) f
( z ) > ∆ / 2 .
If this is true, the reward
Algorithm 6 GuessDelta for n = 1 , 2 , . . .
do ∆ ′
← 12 n
Run Algorithm 2 and Algorithm 3 (equipped with Algorithm 5) with guess margin ∆ ′ and confidence parameter δ n ( n +1) if A policy π is returned from Algorithm 3 then return π end if end for labels of the data points in the data pool is guaranteed to be right, which is enough to guarantee theaccuracy of the learned reward function.
Otherwise we halt the algorithm and try the next guess of ∆ .
The full algorithm is presented in Algorithm 5.
Now we introduce the procedure for guessing ∆ .
We set ∆ ′
= 1 / (2 n ) , ( n = 1 , 2 , ... ) and run
Algorithm 2 and Algorithm 3 repeatedly.
For a guess of ∆ , the output policy is guaranteed to benear-optimal if the algorithms successfully finish and have not been halted by the validation step.
Otherwise, we replace ∆ ′ with ∆ ′ / 2 and rerun the whole algorithm.
The doubling schedule impliesthat the smallest guess is at least ∆ / 2 .
Besides, we also need to adjust the confidence parameter toδ/ ( n ( n + 1)) .
The whole procedure for guessing ∆ is presented in Algorithm 6.
We state the theoretical guarantee in Theorem 4.
Theorem 4.
In the linear case, Algorithm 6 can find an ε -optimal policy with probability at least1 − δ , with at most
O
[Equation]
episodes.
In the tabular case, Algorithm 6 can find an ε -optimal policy with probability at least 1 − δ ,with at most
O
[Equation]
episodes.
In both cases, the total number of queries to the reward is bounded by (cid:101)
O
( H · dim 2 ( F ) / ∆ 2 ) .
Proof.
Note that
[Equation]
Thus we can condition on the good events defined in the proof of Theorem 1 and Theorem 2 for alln ∈ N .
Note that
• With the validation step, the output policy of Algorithm 3 is guaranteed to be ε -optimal,regardless of whether the guess ∆ ′ is true. • Algorithm 3 will output a policy whenever ∆ ′ < ∆ .
As a result, Algorithm 6 will terminate with an ε -optimal policy, and with at most O (log(1 / ∆))guesses of ∆ .
Note that the sample and feedback complexity bounds are both monotonicallyincreasing in 1 / ∆ , thus they at most multiply a log facter log(1 / ∆) .
The difference in the confidenceparameter won’t effect the bound.
F.2 Beyond Binary Reward
In the main paper we assume that the valid reward function is binary.
We remark that our frameworkcan be generalized to RL problems with n -uniform discrete rewards.
We consider a fixed stageh ∈
[ H ] , and omit the subscript h in this section.
In this case, the reward function takes value from { 0 , 1 n , 2 n , . .
In each query, the human teacherchooses from { 0 , 1 n , 2 n , . . . , 1 } (when n = 2 , the choices are { 0 , 12 , 1 } , which can be interpreted as
“bad”, “average”, and “good” actions).
We assume that when queried about a data point z
= ( s, a ) ,the probability of the human teacher choosing in is p
i ( z ) ( 0 ≤ i ≤ n ), and the human response modelf ∗ satisfies:
[Equation]
where f ∗ belongs to the pre-specified function class F .
We assume the true reward of z is determinedby f ∗ ( z ) .
Concretely,
[Equation]
The bounded noise assumption becomes that f ∗ ( z ) can not be too near the decision boundary.
Assumption 4 (Bounded Noise in Uniform Discrete Rewards Setting) .
There exists ∆ > 0 , suchthat for all z ∈ S × A , and all 1 ≤ i ≤ n ,
[Equation]
In Algorithm 1 we estimate the underlying true reward as
[Equation]
The other parts of the algorithm are similar to that with binary rewards.
Following similar analysis inthe proof of Theorem 1, we can learn the reward labels in the data pool correctly using only (cid:101)
O
( d 2 ∆ 2 )queries.
Thus we can derive the exact same sample and feedback complexity bounds as in the binaryreward case.
F.3 Beyond Bounded Noise
In this section we generalize the bounded noise assumption to the low noise assumption (a.k.a, Tsybakov noise)
[Mammen and Tsybakov, 1999, Tsybakov, 2004], which is another standard assumptionin the active learning literature.
Assumption 5 (Low Noise) .
There exists constants α ∈ [0 , 1] and c > 0 , such that for any policy π ,level h ∈
[ H ] , and ε > 0 ,
[Equation]
In this case, the difficulty of the reward learning problem depends on the exponent α .
With this assumption, we can design algorithms with similar feedback and sample complexity.
Concretely, We run Algorithm 2 and Algorithm 3 with ∆ =
( cK ) − 1 α , where K is the number ofepisodes, c
and α are the constants in Assumption 5.
We state the theoretical guarantee in Theorem 5.
Theorem 5.
In the linear case, under Assumption 5, Algorithm 2 and Algorithm 3 with ∆ = ( cK ) − 1 αcan find an ε -optimal policy with probability at least 1 − δ , with at most
K =
O
[Equation]
episodes.
The total number of queries to the reward is bounded by
[Equation]
In the tabular case, under Assumption 5, Algorithm 2 and Algorithm 3 with ∆ = ( cK ) − 1 α can find anε -optimal policy with probability at least 1 − δ , with at most
[Equation]
episodes.
The total number of queries to the reward is bounded by
[Equation]
Proof.
We denote ∆ =
( cK ) − 1 α in the proof.
Let ε = ∆ = ( cK ) − 1 α in Assumption 5.
We have thatfor any policy π , and level h ∈
[ H ] ,
[Equation]
By a martingale version of the Chernoff bound, we have that with probability at least 1 − δ , thenumber of elements in
[Equation]
is at most O (log( H/δ )) for all h ∈
[ H ] .
The active learning algorithm guarantees to learn the reward labels correctly for all the elements inthe dataset D
= { ( s kh , a kh ) } ( h,k ) ∈
[ H ] ×
[ K ] , except the ones in (cid:83)
Hh =1 G h .
Simply rehashing the proofof Theorem 2 shows the optimality of the output policy.
Plugging in the value of ∆ to Theorem 2gives the bound on the total number of queries.