Provably Feedback-Efficient Reinforcement Learning via Active Reward Learning
Anonymous Author(s)
AffiliationAddress
Abstract
An appropriate reward function is of paramount importance in specifying a taskin reinforcement learning (RL).
Yet, it is known to be extremely challenging inpractice to design a correct reward function for even simple tasks.
Human-in-theloop (HiL) RL allows humans to communicate complex goals to the RL agent byproviding various types of feedback.
However, despite achieving great empiricalsuccesses, HiL RL usually requires too much feedback from a human teacher andalso suffers from insufficient theoretical understanding.
In this paper, we focus onaddressing this issue from a theoretical perspective, aiming to provide provablyfeedback-efficient algorithmic frameworks that take human-in-the-loop to specifyrewards of given tasks.
We provide an active-learning -based RL algorithm thatfirst explores the environment without specifying a reward function and then asksa human teacher for only a few queries about the rewards of a task at some stateaction pairs.
After that, the algorithm guarantees to provide a nearly optimal policyfor the task with high probability.
We show that, even with the presence of randomnoise in the feedback, the algorithm only takes (cid:101) O
( H dim R ) queries on the rewardfunction to provide an ε -optimal policy for any ε > 0 .
Here H is the horizonof the RL environment, and dim R specifies the complexity of the function classrepresenting the reward function.
In contrast, standard RL algorithms require toquery the reward function for at least Ω(poly
( d, 1 /ε )) state-action pairs where ddepends on the complexity of the environmental transition.
Introduction
A suitable reward function is essential for specifying a reinforcement learning (RL) agent to performa complex task.
Yet obvious approaches such as hand-designed reward is not scalable for largenumber of tasks, especially in the multitask settings [Wilson et al., 2007, Brunskill and Li, 2013,
Yu et al., 2020, Sodhani et al., 2021] and can also be extremely challenging (e.g., [Ng et al., 1999,
Marthi, 2007] shows even intuitive reward shaping can lead to undesired side effects).
Recently, apopular framework called Human-in-the-loop (HiL) RL [Knox and Stone, 2009, Christiano et al.
,2017, MacGlashan et al., 2017, Ibarz et al., 2018, et al., 2021, Wang et al., 2022] gains moreinterests as it allows humans to communicate complex goals to the RL agent directly by providingvarious types of feedback.
In this sense, a reward function can be learned automatically and can alsobe corrected at proper times if unwanted behavior is happening.
Despite its promising empirical performance, HiL algorithms suffer from insufficient theoretical understanding and possess drawbacks Arakawa al.
[2018], e.g., it assumes humans can give precise numerical rewards and do so without delay and at every time step, which are usually not true.
Moreover, these approaches usually train on every new task separately and cannot incorporate exiting experiences
In this paper, we attempt to address the above issues of incorporating humans’ feedback in RL from atheoretical perspective.
In particular, we would like to address (1) the high feedback complexity issue– i.e., the algorithms in practice usually require large amount feedback from humans to be accurate;(2) feedback from humans can be noisy and non-numerical; (3) in need for support of multiple tasks.
In particular we consider a fixed unknown RL environment, and formulate a task as an unknown butfixed reward function.
A human who wants the agent to accomplish the task needs to communicatethe reward to the agent.
It is not possible to directly specify the parameters of the reward function asthe human may not know it exactly as well, but is able to specify good actions at any given state.
Tocapture the non-numerical feedback issue, we assume that the feedback we can get for an action isonly binary – whether an action is “good” or “bad”.
We further assume that the feedback is noisyin the sense that the feedback is only correct with certain probability.
Lastly, we require that thealgorithm, after some initial exploration phase, should be able to accomplish multiple tasks by onlyquerying the reward rather than the environment again.
In the supervised learning setting, if we only aim to learn a reward function, the feedback complexitycan be well-addressed by the active learning framework Settles [2009], Hanneke et al.
[2014] – analgorithm only queries a few samples of the reward entries and then provide a good estimator.
Yetthis become challenging in the RL setting as it is a sequential decision making problem – state-actionpairs that are important in the supervised learning setting may not be accessible in the RL setting.
Therefore, to apply similar ideas in RL, we need a way to explore the environment and collect samplesthat are important for reward learning.
Fortunately, there were a number of recent works focusing“reward-free” exploration Jin et al.
[2020a], Wang et al.
[2020a] on the environment.
Hence, applyingsuch an algorithm would not affect the feedback complexity.
Additionally it is possible for us to reusethe collected data for multiple tasks.
Our proposed theoretical framework is a non-trivial integration of reward-free reinforcement learningand active learning.
The algorithm possesses two phases: in phase I, it performs reward-free RL toexplore the environment and collect the small but necessary amount of the information about theenvironment; in phase II, the algorithm performs active learning to query the human for the reward atonly a few state-action pairs and then provide a near optimal policy for the tasks with high probability.
The algorithm is guaranteed to work even the feedback is noisy and binary and can solve multipletasks in phase II.
Below we summarize our contributions:
We propose a theoretical framework for incorporating humans’ feedback in RL.
The framework contains two phases: an unsupervised exploration and an active reward learning phase.
Since the two phases are separated, our framework is suitable for multi-task RL.2.
Our framework deals with a general and realistic case where the human feedback is stochasticand binary-i.e., we only ask the human teacher to specify whether an action is “good” or“bad”.
We design an efficient active learning algorithm for learning the reward function fromthis kind of feedback.3.
Our query complexity is minimal because it is independent of both the environmentalcomplexity d and target policy accuracy ε .
In contrast, standard RL algorithms require querythe reward function for at least Ω(poly
( d, 1 /ε )) state-action pairs.
Thus our work providesa theoretical validation for the recent empirical HiL RL works where the number of queriesis significantly smaller than the number of environmental steps.4.
Moreover, we shows the efficacy of our framework in the offline RL setting, where theenvironmental transition dataset is given beforehand.
Related Work
Sample Complexity of Tabular and Linear MDP. There is a long line of theoretical work on thesample complexity and regret bound for tabular MDP. See, e.g., [Kearns and Singh, 2002, Jakschet al., 2010, Azar et al., 2017, Jin et al., 2018, Zanette and Brunskill, 2019, Agarwal et al., 2020, Wang et al., 2020b, Li et al., 2022].
The linear MDP is first studied in Yang and Wang [2019] See,e g. complexity regret Unsupervised Exploration for RL. The reward-free exploration setting is first studied in Jin et al.[2020a].
Wang et al.
[2020a], Zanette et al. [2020b], Kong et al. [2021] further design provably efficient algorithms for reward-free RL with function approximation. Zhang et al.
[2020] studiestask-agnostic RL, which is a variety of reward-free RL.
Wu et al.
[2021] studies multi-objective RLin the reward-free setting.
Active Learning.
Active learning is relatively well-studied in the context of unsupervised learning. See, e.g., Dasgupta et al. [2007], Balcan et al. [2009], Settles [2009], Hanneke et al.
[2014] and thereferences therein.
Our active reward learning algorithm is inspired by a line of works [Cesa-Bianchiet al., 2009, Dekel et al., 2010, Agarwal, 2013] considering online classification problem where theyassume the response model P ( y | x ) is linear parameterized.
However, their works can not directlyapply to the RL setting and also the non-linear case.
Preliminaries3.1 Episodic Markov Decision Process
In this paper, we consider the finite-horizon Markov decision process (MDP) M =
( S , A , P, r, H, s 1 ) ,where S is the state space, A is the action space, P
=
{ P h } Hh =1 where P h : S ×
A → △ ( S ) are thetransition operators, r = { r h } Hh =1 where r h : S
×
A → { 0 , 1 } are the deterministic binary rewardfunctions, and H is the planning horizon.
Without loss of generality, we assume that the initial state s 1is fixed.
1 In RL, an agent interacts with the environment episodically.
Each episode consists of H timesteps.
A deterministic policy π chooses an action a ∈ A based on the current state s ∈ S at each timestep h ∈
[ H ] .
Formally, π = { π h } Hh =1 where for each h ∈
[ H ] , π h : S
→ A maps a given state to anaction.
In each episode, the policy π induces a trajectory s 1 , a 1 , r 1 , s 2 , a 2 , r 2 , ..., s H , a H , r H ,
s H
+1where s 1 is fixed, a 1 = π 1 ( s 1 ) , r 1 = r 1 ( s 1 , a 1 ) , s 2 ∼ P 1 ( ·| s 1 , a 1 ) , a 2 =
π 2 ( s 2 ) , etc.
We use Q-function and V-function to evaluate the long-term expected cumulative reward in termsof the current state (state-action pair), and the policy deployed.
Concretely, the Q-function and
V-function are defined as: Q πh ( s, a ) = E (cid:2) (cid:80)
Hh ′ =
h r h ′
( s h ′ , a h ′ ) | s h
= s, a h
= a, π (cid:3) and V πh ( s ) =
E (cid:2) (cid:80)
Hh ′ =
h r h ′
( s h ′ , a h ′ ) | s h
= s, π (cid:3) .
We denote the optimal policy as π ∗ =
{ π ∗ h } h ∈ [ H ] , optimalvalues as Q ∗ h ( s, a ) and V ∗ h ( s ) .
Sometimes it is convenient to consider the Q-function and V-functionwhere the true reward function is replaced by a estimated one ˆ r = {
ˆ r h } h ∈
[ H ] .
We denote themas Q πh ( s, a, ˆ r ) and V πh ( s, ˆ r ) .
We also denote the corresponding optimal policy and value as π ∗ (ˆ r ) ,
Q ∗ h ( s, a, ˆ r ) and V ∗ h ( s, ˆ r ) .
Additional Notations.
We define the infinity-norm of function f : S ×
A → R as ∥ f ∥ ∞ =sup ( s,a ) ∈S×A | f
( s, a ) | .
For a set of state-action pairs Z ⊆ S
× A and a function f : S ×
A → R ,we define ∥ f ∥
Z = (cid:16)(cid:80) ( s,a ) ∈S×A
f ( s, a ) 2 (cid:17) 1 / 2 .
Technical Overview
[Figure]
In this section we give a overview of our learning scenario and notations, as well as the maintechniques.
The learning process divides into three phases.
4.1 Phase 1: Unsupervised Exploration
The first step is to explore the environment without reward signals.
Then we can query the humanteacher about the reward function in the explored region.
We adopt the reward-free explorationtechnique developed in Jin et al.
[2020a], Wang et al.
[2020a].
The agent is encouraged to doexploration by maximizing the cumulative exploration bonus .
Concretely, we gather K trajectories
D = { ( s kh , a kh ) } ( h,k ) ∈
[ H ] ×
[ K ] by interacting with the environment.
We can strategically choose whichpolicy to use.
At the beginning of the k -th episode, we calculate a policy π k based on the history ofthe first k − 1 episodes and use π k to induce a trajectory { ( s kh , a kh ) } h ∈ [ H ] .
A similar approach called unsupervised pre-training [Sharma et al., 2020, Liu and Abbeel, 2021] hasbeen successfully used in practice.
Concretely, in unsupervised pre-training agents are encouraged to
For a general initial distribution ρ , we can treat it as the first stage transition probability, P 1 .
do exploration by maximizing various intrinsic rewards , such as prediction errors [Houthooft et al. and count-based state-novelty [Tang et al., 2017].
4.2 Phase 2: Active Reward Learning
The second step is to learn a proper reward function from human feedback.
Our work assumes that theunderlying valid reward is 1-0 binary, which is interpreted as good action and bad action.
Concretely,consider a fixed stage
h ∈
[ H ] , and we are trying to learn r h from the human response.
Each timewe can query a datum z
= ( s, a ) and receive an independent random response Y ∈ { 0 , 1 } from thehuman expert, with distribution:
[Equation]
Here f ∗ h is the human response model and needs to be learned from data.
We assume that theunderlying valid reward of z can be determined by f ∗ h ( z ) in the following manner:
[Equation]
Note that the query returns 1 with a probability greater than 12 if and only if the underlying validreward is 1.
To make the number of queries as small as possible, we choose a small subset ofinformative data to query the human.
We adopt ideas in the pool-based active learning literature andselect informative queries greedily.
We show that only (cid:101) O
( H dim R ) queries need to be answered bythe human teacher.
The active query method is widely used in human-involved reward learning inpractice and shows superior performance than uniform sampling Christiano et al.
[2017], Ibarz et al.[2018], Lee et al.
After we learn a proper reward function ˆ r , we use past experience D and ˆ r to plan for a good policy.
Note that in this phase we are not allowed for further interaction with the environment.
In themulti-task RL setting, we can run Phase 2 for multiple times and reuse the data collected in Phase 1.
Now we discuss the efficacy of our framework.
A naive approach for reward learning via humanfeedback is asking the human teacher to evaluate the reward function in each round.
This approachresults in equal environmental steps and number of queries.
This high query frequency is unacceptablefor large-scale problems.
For example, in Lee et al.
[2021] the agent learns complex tasks with veryfew queries ( ∼ 10 2 to 10 3 queries) to the human compared to the number of environmental steps( ∼ 10 6 steps) by utilizing active query technique.
From the theoretical perspective, usual RL samplecomplexity bound scales with ∝ poly( d, 1 ε ) , where d is the complexity measure of the environmentaltransition and ε is the target policy accuracy.
This quantity can be huge when the environment iscomplex (i.e., d is large) or with small target accuracy.
Our query complexity is desirable since it isindependent of both d and 1 /ε .
Pool-Based Active Reward Learning
In this section we formally introduce our algorithm for active reward learning.
We consider a fixedstage h and learn r h by querying a small subset of Z h
= { ( s kh , a kh ) } k ∈
[ K ] .
We omit the subscripth in this section, i.e., we use Z , z k , r, f ∗ to denote Z h , z kh , r h , f ∗ h in this section.
Since Z is givenbefore the learning process starts, we refer to this learning scenario as pool-based active learning.
Our purpose is to learn a reward function
ˆ r ( · ) such that r ( z ) =
ˆ r ( z ) for most of z in Z .
At the sametime, we hope the number of queries can be as small as possible.
We assume F is a pre-specified function class to learn f ∗ from, and F is known as a prior.
Weassume that F has enough expressive power to represent the human response.
Concretely, we assumethe following realizability .
Assumption 1 (Realizability) .
f ∗ ∈ F
The learning problem can be arbitrarily difficult, especially when f ∗ ( z ) is close to 1 2 , in which caseit will be difficult to determine the true value of r ( z ) .
However, the marginal case is rare in manyreal-world problems, and the learning goal is not that difficult to identify for human beings.
To give aproblem-dependent bound, we assume the following bounded noise assumption.
In the literature onstatistical learning, this assumption is also referred to as Massart noise [Massart and Nédélec, 2006, Giné and Koltchinskii, 2006, Hanneke et al., 2014].
Assumption 2 (Bounded Noise) .
There exists ∆ > 0 , such that for all z ∈ S × A , | f ∗ ( z ) − 12 | > ∆ .
Examples We give two examples of F that is frequently studied in the active learning literature.
Inthe linear model, the function class F consists of f in the following form:
[Equation]
In the logistic model, the function class F consists of f in the following form:
[Equation]
Here ϕ : S ×
A → R d is a fixed and known feature extractor, and w ∈ R d .
The complexity of F essentially depends on the learning complexity of the human response model.
We use the following Eluder dimension [Russo and Van Roy, 2014] to characterize the complexity of
F .
The eluder dimension serves as a common complexity measure of a general non-linear functionclass in both reinforcement learning literature [Osband and Van Roy, 2014, Ayoub et al., 2020, Wanget al., 2020c, Jin et al., 2021a] and active learning literature [Chen et al., 2021].
Definition 1 (Eluder Dimension) .
Let ε ≥ 0 and Z
= { ( s i , a i ) } ni
=1 ⊆ S ×
A be a sequence ofstate-action pairs.
(1) A state-action pair ( s, a ) ∈ S
× A is ε -dependent on Z with respect to F if any f, f ′ ∈ F
satisfying ∥
f − f ′
∥
Z ≤ ε also satisfies | f ( s, a )
− f ′ ( s, a ) | ≤ ε .
(2) An ( s, a ) is ε -independent of Z with respect to F if ( s, a ) is not ε -dependent on Z .
(3) The ε -eluder dimension dim E ( F , ε ) of a function class F is the length of the longest sequence of elements in S ×
A such that, for some ε ′ ≥ ε , every element is ε ′ -independent of its predecessors.(4)
The eluder dimension of a function class F is defined as
[Equation]
We remark that a wide range of function classes, including linear functions, generalized linearfunctions and bounded degree polynomials, have bounded eluder dimension.
Definition 2 (Covering Number and Kolmogorov Dimension) .
For any ε > 0 , there exists an ε -cover
C ( F , ε ) ⊆
F with size |C ( F , ε ) | ≤ N ( F , ε ) , such that for any f ∈ F , there exists f ′ ∈ C ( F , ε ) with∥
f − f ′ ∥ ∞ ≤ ε .
The Kolmogorov dimension of F is defined as:
[Equation]
The Kolmogorov dimension is also bounded by O ( d ) for linear/generalized linear function class.
Throughout this paper, we denote
[Equation] as the complexity measure of F .
When F is the class of d -dimensional linear/generalized linearfunctions, dim( F ) is bounded by O ( d ) .
5.1 Algorithm
We describe our algorithm for learning the human response model and the underlying reward function.
We sequentially choose which data points to query.
Denote Z k the first k points that we decide toquery and initial Z 0 to be an empty set.
For each z ∈ Z , we use the following bonus function tomeasure the information gain of querying z , i.e., how much new information z contains compared to
Z k − 1 :
[Equation]
We then simply choose z k to be arg max z ∈Z b k ( z ) .
After the N query points are determined, wequery their labels from a human.
The human response model is then learned by solving a least-squaresregression:
[Equation]
∈Z
[Equation] f ( z ) − l ( z )) The human response model is used for estimating the underlying reward function.
The wholealgorithm is presented in Algorithm 1.
Note that such an interactive mode with the human teacher isnon-adaptive since all queries are given to the human teacher in one batch.
This property makes ouralgorithm desirable in practice.
Algorithm
Input: Data Pool Z = { z i } i ∈
[ T ] , margin ∆ , failure probability δ ∈ (0 , 1)
Z 0 ← {} //Query
Dataset
Set N ← C 1 · (dim 2 ( F )+dim( F ) · log(1 /δ ))
· (log 2 (dim( F )))
∆for k = 1 , 2 , ..., N do β ← C 2 · (cid:112) log(1 /δ )
+ log N · dim( F )
Set the bonus function: b k ( · ) ← sup f,f ′ ∈F , ∥ f − f ′ ∥
Z k − 1 ≤ β | f ( · ) − f ′ ( · ) | z k ← arg max z ∈Z b k ( z )
Z k ← Z k − 1 ∪ { z k } end for for z ∈ Z N do
Ask the human expert for a label l ( z ) ∈ { 0 , 1 } end for
Estimate the human model as (cid:101) f = arg min f ∈F (cid:80) z ∈Z N ( f ( z ) − l ( z ))
Let ˆ f ∈ C ( F , ∆ / 2) such that ∥ ˆ f − (cid:101) f ∥ ∞ ≤ ∆ /
Estimate the underlying true reward: ˆ r ( · ) =
(cid:40) 1 , ˆ f ( · ) > 1 /0 , ˆ f ( · ) ≤ 1 /return:
The estimated reward function ˆ r .
5.2 Theoretical Guarantee
Theorem 1.
With probability at least 1 − δ , for all z ∈ Z , we have, ˆ r ( z ) =
r ( z ) .
The total numberof queries is bounded by
[Equation]
Proof Sketch The first step is to show that the sum of bonus functions (cid:80)
Kk
=
b k ( z k ) is boundedby O ( d √ K ) using ideas in Russo and Van Roy [2014].
Note that the bonus function b k ( · ) is nonincreasing.
Thus we can show that after selecting N = (cid:101) O
( d 2 ∆ 2 ) points, for all z ∈ Z , the bonusfunction of z does not exceed ∆ .
By the low-noise assumption, we know that the reward label for zis correct for all z .
Online RL with Active Reward Learning
In this section we consider how to apply active reward learning method in the online RL setting.
Inthis setting the agent is allowed to actively explore the environment without reward signal in theexploration phase.
We consider both tabular MDP and linear MDP cases.
6.1 Linear MDP with Positive Features
The linear MDP assumption was first studied in Yang and Wang [2019] and then applied in the onlinesetting Jin et al.
[2020b].
It is assumed that the agent is given a feature extractor ϕ : S ×
A → R dand the transition model can be predicted by linear functions of the give feature extractor.
In ourwork, we additionally assume that the coordinates of ϕ are all positive .
This assumption essentiallyreduce the linear MDP model to the soft state aggregation model [Singh et al., 1994, Duan et al.
As will be seen later, the latent state structure helps the learned reward function to generalize.
Assumption 3 (Linear MDP with Non-Negative Features) .
For all h ∈ [ H ] , we assume thatthere exists a function µ h : S → R d such that P h ( s ′ | s, a ) =
⟨ µ h ( s ′ ) , ϕ ( s, a ) ⟩ .
Moreover, for all( s, a ) ∈ S × A , the coordinates of ϕ ( s, a ) and µ h ( s, a ) are all non-negative.
6.2 Exploration Phase
In the exploration phase, inspired by former works on reward-free RL, we use optimistic least-squaresvalue iteration (LSVI) based algorithm with zero reward.
In the linear case, for any V : S → R , weestimate P h V in the following manner
[Equation]
P
[Equation]
[Equation] ϕ [Equation] · , · ) , where w ← arg min
[Equation]
For the tabular case, we simply use the empirical estimation of P h :
[Equation]
and define (cid:98) P kh V in the conventional manner.
Here N kh
( s, a, s ′ )
= (cid:80) k − 1 τ =1 1 { ( s τh , a τh , s τh +1 ) =(
s, a, s ′ ) } and N kh ( s, a ) =
(cid:80)
k − 1 τ =1 1 { ( s τh , a τh ) =
( s, a ) } are the numbers of visit time.
The following optimism bonus Γ kh ( · , · ) is sufficient to guarantee optimism in standard regret minimization RL algorithms.
(The choices of β tbl and β lin is specified in the appendix) [Equation] Tabular
[Equation]
In our setting we enlarge the optimism bonus to the following exploration bonus .
[Equation]
Tabular Case [Equation]
[Equation]
We then set the optimistic Q-function as
[Equation]
and define the exploration policy as the greedy policy with respect to Q kh .
Algorithm for k = 1 , 2 , ..., K do
V kH +1 ← 0 , Q kH +1 ←for h
= H, H − 1 , ..., 1 do
Estimate (cid:98) P kh V kh ( · , · ) using (1) or (2)
[-+
 ( · , · ) using (1) or (2)-] Set the optimism bonus Γ kh ( · , · ) using (3)
Set the exploration bonus b k h ( · , · ) using (4).
Set the optimistic Q-function Q kh ( · , · ) ← Π [0 ,H − h +1]
[ (cid:98) P kh V kh ( · , · ) + b kh ( · , · )] π kh ( · ) ← arg max a ∈A Q kh ( · , a )
V kh ( · ) ← max a ∈A Q kh ( · , a ) end for
Execute policy π k = { π kh } h ∈
[ H ] to induce a trajectory s k 1 , a k 1 , ..., s kH , a kH , s kH +1 .
end for return: Dataset D = { ( s kh , a kh ) }
( h,k ) ∈
[ H ] ×
[ K ]
6.3 Reward Learning & Planning Phase
After the exploration phase, we run the active reward learning algorithm introduced before on thecollected dataset.
In the linear setting, we replace the original action with uniform random action.
Wethen use the learned reward function to plan for a near-optimal policy.
We still add optimism bonusto guarantee optimism.
The whole algorithm is presented in Algorithm
Algorithm
Input: Dataset D = { ( s kh , a kh ) } ( h,k ) ∈
[ H ] ×
[ K ] for h = 1 , 2 , ..., H do if Linear Case then (cid:101)
Z h ← { ( s kh , (cid:101) a kh ) } k ∈ [ K ] , where { (cid:101) a kh } k ∈
[ K ] are sampled i.i.d.
from Unif( A )
ˆ r h ← Active Reward Learning ( (cid:101)
Z h , ∆ , δ/ (2 H )) .
else if Tabular Case then
Z h ← { ( s kh , a kh ) } k ∈
[ K ] ˆ r h ← Active Reward Learning ( Z h , ∆ , δ/ (2 H )) .
end if end for for k = 1 , 2 , ...,
K do
V kH +1 ← 0 , Q kH +1 ← 0 for h = H, H − 1 , ..., 1 do
Estimate (cid:98) P kh V kh ( · , · ) using (1) or (2)
[-+
 ( · , · ) using (1) or (2)-] Set the optimism bonus Γ kh ( · , · ) using (3)
Set the optimistic Q-function Q kh ( · , · ) ← Π [0 ,H − h +1]
[ˆ r h ( · , · ) + (cid:98) P
h V kh[-+
-] + Γ kh ( · , · )]
ˆ π kh ( · ) ← arg max a ∈A Q kh ( · , a )
V kh ( · ) ← max a ∈A Q kh ( · , a ) end for end for return: ˆ π drawn uniformly from { ˆ π k } Kk =1 where π k = { ˆ π kh } h ∈
[ H ]
6.4 Theoretical Guarantee
Theorem 2.
In the linear case, our algorithm can find an ε -optimal policy with probability at least1 − δ , with at most O (cid:18) [Equation] [Equation]
ε
[Equation]
(cid:19)
[Equation]
) episodes.
In the tabular case, our algorithm can find an ε -optimal policy with probability at least1 − δ , with at most
[Equation] episodes. In both cases, the total number of queries to the reward is bounded by (cid:101)
O
( H · dim 2 ( F ) / ∆ 2 ) .
Remark 1.
Theorem 2 readily extends to multi-task RL setting by replacing δ with δ/N and applyinga union bound over all tasks, where N is the number of tasks.
The corresponding sample complexitybound only increase by a factor of poly log( N ) .
Proof Sketch The suboptimality of the policy ˆ π can be decomposed into two parts:
[Equation]
where (i) correspond to the planning error in the planning phase (Algorithm 3) and (ii) correspondto the estimation error of the reward ˆ r .
By standard techniques from the reward-free RL, (i) canbe upper bounded by the expected summation of the exploration bonuses in the exploration phase.
In order to bound (ii), we need the learned reward function to be universally correct, not just onthe explored region.
We show that the dataset collected in the exploration phase essentially coverthe state space (tabular case) or the latent state space.
Since the reward function class has boundedcomplexity ( log |R| is bounded due to the bounded covering number of F ), the reward functionlearned from the exploratory dataset can generalized to a distribution induced by any policy.
Offline RL with Active Reward Learning
In this section we consider the offline RL setting, where the dataset D is provided beforehand.
Weshow that our active reward learning algorithm can still work well in this setting.
In order to givemeaningful result, we assume the following compliance property of D with respect to the underlying
MDP.
This assumption is firstly introduced in Jin et al.
[2021b].
Unlike many literature for offline
RL, we do not require strong coverage assumptions, e.g., concentratability [Szepesvári and Munos,2005, Antos et al., 2008, Chen and Jiang, 2019].
[-P D ( s kh +
-]
Definition 3 (Compliance) .
For a dataset D = { ( s kh , a kh ) } ( h,k ) ∈
[ H ] ×
[ K ] , let P D be the joint distribu- 284 tion of the data collecting process.
We say D is compliant with the underlying MDP if
[Equation]
[-For any f ∈ F and z ∈ Z N , considerξ ( z, f ) = 
-]
holds for all h ∈
[ H ] , k ∈
[ K ] , s ∈ S .
7.1 Algorithm and Theoretical Guarantee
At the beginning of the algorithm we call the active reward learning algorithm to estimate the rewardfunction.
Inspired by Jin et al.
[2021b], we estimated the optimal Q-value Q k using pessimistic valueiteration with empirical transition and learned reward function.
The policy is defined as the greedypolicy with respect to Q k .
The full algorithm and theoretical guarantee is stated below.
Input: Dataset D = { ( s kh , a kh ) } ( h,k ) ∈
[ H ] ×
[ K ] for h = 1 , 2 , ..., H do
Z h ← { ( s kh , a kh ) } k ∈
[-+
-][ K ] ˆ r h ← Active Reward Learning ( Z h , ∆ , δ/ (2 H )) .
[-kh +
-]
end for (cid:98) V [-kh +
 ( s, a ) = ϕ ( s, a ) T (cid:101) w kh-]H +1 ← 0 .
for h = H, H − 1 , ..., 1 do Γ h ( · , · ) ← β ′ tbl · ( N h ( · , · ) + 1) − 1 /
Q h ( · , · ) ← Π [0 ,H − h +1]
[ˆ r h ( · , · ) + (cid:98) P h (cid:98) V
h
+1 ( · , · ) − 2Γ h ( · , · )]
ˆ π h ( · ) ← arg max a ∈A Q h ( · , a )
V h ( · ) ← max a ∈A Q h ( · , a ) end for return: ˆ π = { ˆ π h } h ∈
[ H ]
Theorem 3.
With probability at least 1 − δ , the sub-optimal gap of ˆ π is bounded by
V
[Equation]
s
[Equation]
(cid:32)
H (cid:112)
[Equation]
( SAHK/
[Equation]
[Equation]
E
[Equation]
∗ (cid:34)
[Equation]
[-+ 
-]
(cid:88) h
= ( N h ( s h , a h ) + 1) − 1 /(cid:35)(cid:33) .
And the total number of queries is bounded by (cid:101)
O ( H · dim 2 ( F ) / ∆ 2 ) .
The proof of Theorem 3 is deferred to the appendix.
Conclusion
In this work, we provide a provably feedback-efficient algorithmic framework that takes human-in-theloop to specify rewards of given tasks.
Our proposed framework theoretically addresses several issuesof incorporating humans’ feedback in RL, such as noisy, non-numerical feedback and high feedbackcomplexity.
Technically, our work integrates reward-free RL and active learning in a non-trivial way.
We hope our work provides a theoretical validation for recent empirical successes in HiL RL andcan inspire future HiL RL algorithm design.
An interesting future direction is incorporating differentactive learning methods, such as disagreement-based active learning, into our framework.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope? [Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 8 for future directions. (c) Did you discuss any potential negative societal impacts of your work? [N/A] Thiswork is theoretical in nature and does not have immediate social impact.(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results? [Yes] See Assumptions 1,2, (b) Did you include complete proofs of all theoretical results?
[Yes] Proof of all theoremsand lemmas are included in the Supplementary Material. If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
[N/A]
(c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] If you are using existing assets (e
g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]